[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:12 dist-test-slave-mcnm krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:12 dist-test-slave-mcnm krb5kdc[395](info): set up 2 sockets
Jun 20 18:29:12 dist-test-slave-mcnm krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:29:13.392086 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:13 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:13.564289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44119
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:13.888018 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:13.888392 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:13.888564 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:13.896221 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:13.896385 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:13.896490 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:13.896597 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:13.896677 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:13.906390 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:13.907835 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:13.909188 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:13.921759 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:13.921828 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:13.921782 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:13.922652 408 server_base.cc:1034] running on GCE node
I20260620 18:29:13.923641 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:13.925247 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:13.926501 408 hybrid_clock.cc:648] HybridClock initialized: now 1781980153926431 us; error 79 us; skew 500 ppm
I20260620 18:29:13.926904 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:13 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:13.930647 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:13.932284 408 webserver.cc:460] Webserver started at http://127.0.97.62:36257/ using document root <none> and password file <none>
I20260620 18:29:13.932956 408 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:13.933051 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:13.933297 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:13.936283 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "7b4deb4a97f24306940d9b10e000bf4f"
format_stamp: "Formatted at 2026-06-20 18:29:13 on dist-test-slave-mcnm"
server_key: "d4b3211c344d54d25b455f7f40983968"
server_key_iv: "2e8f8207926c074529a2b1adcfbfd162"
server_key_version: "encryptionkey@0"
I20260620 18:29:13.937011 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b4deb4a97f24306940d9b10e000bf4f"
format_stamp: "Formatted at 2026-06-20 18:29:13 on dist-test-slave-mcnm"
server_key: "d4b3211c344d54d25b455f7f40983968"
server_key_iv: "2e8f8207926c074529a2b1adcfbfd162"
server_key_version: "encryptionkey@0"
I20260620 18:29:13.942328 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:29:13.945894 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:13.947280 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:29:13.947479 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "7b4deb4a97f24306940d9b10e000bf4f"
format_stamp: "Formatted at 2026-06-20 18:29:13 on dist-test-slave-mcnm"
server_key: "d4b3211c344d54d25b455f7f40983968"
server_key_iv: "2e8f8207926c074529a2b1adcfbfd162"
server_key_version: "encryptionkey@0"
I20260620 18:29:13.947753 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:13.984890 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:13.989480 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:13.989816 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:14.009945 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44119
I20260620 18:29:14.009970 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44119 every 8 connection(s)
I20260620 18:29:14.011482 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:14.016052 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:14.020905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 408
I20260620 18:29:14.021209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:14.023008 388 external_mini_cluster.cc:1428] Setting key fe990b361e677ef8716f75556ab21342
I20260620 18:29:14.025605 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f: Bootstrap starting.
I20260620 18:29:14.030133 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:14.031260 475 log.cc:824] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:14.034349 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f: No bootstrap required, opened a new log
I20260620 18:29:14.045727 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } }
I20260620 18:29:14.046236 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:14.046339 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4deb4a97f24306940d9b10e000bf4f, State: Initialized, Role: FOLLOWER
I20260620 18:29:14.046996 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } }
I20260620 18:29:14.047262 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:14.047937 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:14.048169 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:14.050809 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } }
I20260620 18:29:14.051390 475 leader_election.cc:302] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4deb4a97f24306940d9b10e000bf4f; no voters:
I20260620 18:29:14.052150 475 leader_election.cc:288] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:14.052317 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:14.053071 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [term 1 LEADER]: Becoming Leader. State: Replica: 7b4deb4a97f24306940d9b10e000bf4f, State: Running, Role: LEADER
I20260620 18:29:14.053620 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } }
Jun 20 18:29:14 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:14.055964 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:14.056743 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b4deb4a97f24306940d9b10e000bf4f. Latest consensus state: current_term: 1 leader_uuid: "7b4deb4a97f24306940d9b10e000bf4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } } }
I20260620 18:29:14.057150 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:14.058168 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b4deb4a97f24306940d9b10e000bf4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4deb4a97f24306940d9b10e000bf4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44119 } } }
I20260620 18:29:14.058444 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:14.067034 484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:14.068336 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:14.027292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.97.62:44119)
0620 18:29:14.028437 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:14.028458 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:14.029662 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 18:29:14.040015 (+ 10353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:14.040044 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:14.040598 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:14.041104 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:14.043739 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.045974 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.048168 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.048897 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.049114 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:14.055954 (+ 6840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:14.055961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:14.055974 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:14.056008 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:14.061109 (+ 5101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:14.062377 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:14.062390 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:14.062401 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:14.062656 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:14.063313 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:14.063321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:14.063328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:14.064173 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:14.064550 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:14.066056 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0620 18:29:14.066502 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":595,"threads_started":1}
I20260620 18:29:14.072655 484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:14.082904 484 catalog_manager.cc:1347] Generated new cluster ID: 7a8c0ba60d454e32a8bba8c0ea071d80
I20260620 18:29:14.083029 484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:14.124311 484 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:14.125635 484 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:14.135854 484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b4deb4a97f24306940d9b10e000bf4f: Generated new TSK 0
I20260620 18:29:14.136700 484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:14.187247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:14.518443 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:14.518719 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:14.518889 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:14.526664 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:14.526783 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:14.526959 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:14.536955 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:14.538239 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:14.539603 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:14.553387 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:14.553414 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:14.553340 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:14.554629 501 server_base.cc:1034] running on GCE node
I20260620 18:29:14.555215 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:14.556260 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:14.557479 501 hybrid_clock.cc:648] HybridClock initialized: now 1781980154557425 us; error 46 us; skew 500 ppm
I20260620 18:29:14.557793 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:14 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:14.561296 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:14.562796 501 webserver.cc:460] Webserver started at http://127.0.97.1:35303/ using document root <none> and password file <none>
I20260620 18:29:14.563459 501 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:14.563617 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:14.564049 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:14.567049 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "99aae1c0934943b6a3b0557857812346"
format_stamp: "Formatted at 2026-06-20 18:29:14 on dist-test-slave-mcnm"
server_key: "357398e26060570237abca544bc3ca62"
server_key_iv: "7f4d39b0b3762df847f5e70760630cad"
server_key_version: "encryptionkey@0"
I20260620 18:29:14.567910 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "99aae1c0934943b6a3b0557857812346"
format_stamp: "Formatted at 2026-06-20 18:29:14 on dist-test-slave-mcnm"
server_key: "357398e26060570237abca544bc3ca62"
server_key_iv: "7f4d39b0b3762df847f5e70760630cad"
server_key_version: "encryptionkey@0"
I20260620 18:29:14.573076 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:29:14.576433 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:14.577764 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:29:14.577932 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "99aae1c0934943b6a3b0557857812346"
format_stamp: "Formatted at 2026-06-20 18:29:14 on dist-test-slave-mcnm"
server_key: "357398e26060570237abca544bc3ca62"
server_key_iv: "7f4d39b0b3762df847f5e70760630cad"
server_key_version: "encryptionkey@0"
I20260620 18:29:14.578132 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:14.604907 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:14.609366 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:14.609669 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:14.610628 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:14.611999 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:14.612102 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:14.612253 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:14.612326 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:14.649389 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43717
I20260620 18:29:14.649420 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43717 every 8 connection(s)
I20260620 18:29:14.650983 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:14.654035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 501
I20260620 18:29:14.654253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:14.654856 388 external_mini_cluster.cc:1428] Setting key 1f59b2c84a4a7d281d81e07e61e9e048
Jun 20 18:29:14 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:29:14.674326 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:14.653797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52115 (local address 127.0.97.62:44119)
0620 18:29:14.654334 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:14.654341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:14.655377 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 18:29:14.657866 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:14.657874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:14.657966 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:14.658092 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:14.659917 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.661134 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.663790 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.664289 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.664428 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:14.668620 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:14.668634 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:14.668638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:14.668669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:14.671161 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:14.671964 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:14.671970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:14.671976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:14.672106 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:14.672528 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:14.672533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:14.672536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:14.672798 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:14.672956 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:14.673739 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 18:29:14.673955 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":246,"threads_started":1}
I20260620 18:29:14.675248 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:14.654368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44119 (local address 127.0.97.1:52115)
0620 18:29:14.655203 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:14.655250 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:14.657594 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:14.658363 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:14.658372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:14.658963 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:14.659719 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:14.659742 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.661315 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.661321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:14.663611 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:14.663629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:14.664533 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:14.664543 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:14.664885 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:14.665680 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:14.665715 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:14.668399 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:14.671324 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:14.671332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:14.671348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:14.671837 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:14.672224 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:14.672231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:14.672234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:14.672406 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:14.672954 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:14.672963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:14.673307 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 18:29:14.673646 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 18:29:14.674149 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":335,"threads_started":1}
I20260620 18:29:14.676831 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44119
I20260620 18:29:14.677234 629 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:14.677963 629 heartbeater.cc:511] Master 127.0.97.62:44119 requested a full tablet report, sending...
I20260620 18:29:14.680353 440 ts_manager.cc:194] Registered new tserver with Master: 99aae1c0934943b6a3b0557857812346 (127.0.97.1:43717)
I20260620 18:29:14.682497 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:14.699316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:15.035408 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:15.035802 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:15.035964 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:15.045425 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:15.045542 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:15.045666 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:15.057133 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:15.058454 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:15.059873 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:15.073278 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:15.073493 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:15.073434 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:15.074887 635 server_base.cc:1034] running on GCE node
I20260620 18:29:15.075714 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:15.076793 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:15.078022 635 hybrid_clock.cc:648] HybridClock initialized: now 1781980155077981 us; error 33 us; skew 500 ppm
I20260620 18:29:15.078308 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:15.081532 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:15.083104 635 webserver.cc:460] Webserver started at http://127.0.97.2:43175/ using document root <none> and password file <none>
I20260620 18:29:15.083876 635 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:15.084035 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:15.084365 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:15.087332 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "5c05ae9cb933454497fc699f0cd6d02c"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "71abc07a03ac76572c4ca5b0adae4e0a"
server_key_iv: "bf10bf73930811d951f7208baae269c3"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.088238 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5c05ae9cb933454497fc699f0cd6d02c"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "71abc07a03ac76572c4ca5b0adae4e0a"
server_key_iv: "bf10bf73930811d951f7208baae269c3"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.093532 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:29:15.097100 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.098590 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:29:15.098766 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "5c05ae9cb933454497fc699f0cd6d02c"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "71abc07a03ac76572c4ca5b0adae4e0a"
server_key_iv: "bf10bf73930811d951f7208baae269c3"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.098932 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:15.112690 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:15.116852 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:15.117200 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:15.118207 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:15.119495 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:15.119633 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.119791 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:15.119858 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.156124 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32921
I20260620 18:29:15.156155 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32921 every 8 connection(s)
I20260620 18:29:15.157451 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:15.165891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 635
I20260620 18:29:15.166095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:15.166538 388 external_mini_cluster.cc:1428] Setting key 5b81ea5029865c7d06668f9a87846420
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:15.179796 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.160129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53399 (local address 127.0.97.62:44119)
0620 18:29:15.160297 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:15.160302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:15.161498 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 18:29:15.163270 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:15.163275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:15.163343 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:15.163432 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:15.165173 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.165905 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.167931 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.168554 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.168686 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.172806 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:15.172816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:15.172823 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:15.172872 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:15.176227 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.177270 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.177274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.177276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.177420 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.177977 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.177981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.177983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.178259 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:15.178367 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:15.179320 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 18:29:15.179487 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:29:15.181452 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.160665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44119 (local address 127.0.97.2:53399)
0620 18:29:15.161332 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:15.161374 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:15.162996 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:15.163903 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:15.163911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:15.164447 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:15.164996 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:15.165012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.166117 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.166121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:15.167767 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:15.167780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.168763 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.168768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:15.168963 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.169701 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:15.169724 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:15.172543 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:15.176395 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:15.176407 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:15.176429 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:15.177111 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:15.177570 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:15.177578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:15.177585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:15.177852 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:15.178390 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:15.178401 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:15.178898 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 18:29:15.179220 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 18:29:15.179880 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":258,"threads_started":1}
I20260620 18:29:15.183089 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44119
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:29:15.183444 763 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:15.184159 763 heartbeater.cc:511] Master 127.0.97.62:44119 requested a full tablet report, sending...
I20260620 18:29:15.185707 440 ts_manager.cc:194] Registered new tserver with Master: 5c05ae9cb933454497fc699f0cd6d02c (127.0.97.2:32921)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 18:29:15.186731 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:15.207594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:15.529316 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:15.529630 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:15.529789 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:15.536945 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:15.537091 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:15.537269 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:15.547116 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:15.548455 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:15.549780 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:15.564059 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:15.564087 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:15.564146 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:15.565577 768 server_base.cc:1034] running on GCE node
I20260620 18:29:15.566325 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:15.567488 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:15.568849 768 hybrid_clock.cc:648] HybridClock initialized: now 1781980155568789 us; error 65 us; skew 500 ppm
I20260620 18:29:15.569204 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:15.572597 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:15.574191 768 webserver.cc:460] Webserver started at http://127.0.97.3:42339/ using document root <none> and password file <none>
I20260620 18:29:15.574892 768 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:15.575035 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:15.575342 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:15.578366 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "e216be957abb467ab85f1919f063f407"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "d914addc6bcd979f9b820b596162d952"
server_key_iv: "61cc16e946b8203c18d36359b3f69475"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.579128 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e216be957abb467ab85f1919f063f407"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "d914addc6bcd979f9b820b596162d952"
server_key_iv: "61cc16e946b8203c18d36359b3f69475"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.584758 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:29:15.588132 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.589452 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:29:15.589624 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "e216be957abb467ab85f1919f063f407"
format_stamp: "Formatted at 2026-06-20 18:29:15 on dist-test-slave-mcnm"
server_key: "d914addc6bcd979f9b820b596162d952"
server_key_iv: "61cc16e946b8203c18d36359b3f69475"
server_key_version: "encryptionkey@0"
I20260620 18:29:15.589825 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:15.605501 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:15.609687 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:15.609972 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:15.610919 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:15.612371 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:15.612470 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.612618 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:15.612692 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:15.649498 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36927
I20260620 18:29:15.649538 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36927 every 8 connection(s)
I20260620 18:29:15.650794 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:15.653190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 768
I20260620 18:29:15.653366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:15.653863 388 external_mini_cluster.cc:1428] Setting key f33e87f641e7bdb5b1a821734b48f378
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:15.672192 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.653547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53225 (local address 127.0.97.62:44119)
0620 18:29:15.653743 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:15.653750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:15.655016 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 18:29:15.657418 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:15.657424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:15.657513 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:15.657620 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:15.659130 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.660063 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.661998 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.662398 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.662525 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.665878 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:15.665883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:15.665886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:15.665916 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:15.669062 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.669890 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.669896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.669902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.670043 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.670484 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.670490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.670493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.670760 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:15.670905 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:15.671622 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 18:29:15.671829 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:29:15.673118 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.654047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44119 (local address 127.0.97.3:53225)
0620 18:29:15.654856 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:15.654899 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:15.657113 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:15.657932 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:15.657944 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:15.658436 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:15.658985 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:15.659002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.660212 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.660216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:15.661859 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:15.661871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.662597 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.662602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:15.662804 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.663427 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:15.663468 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:15.665680 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:15.669222 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:15.669229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:15.669243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:15.669774 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:15.670162 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:15.670169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:15.670173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:15.670370 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:15.670882 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:15.670890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:15.671231 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 18:29:15.671497 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:29:15.672050 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":339,"threads_started":1}
I20260620 18:29:15.674533 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44119
I20260620 18:29:15.674854 896 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:15.675658 896 heartbeater.cc:511] Master 127.0.97.62:44119 requested a full tablet report, sending...
I20260620 18:29:15.677196 440 ts_manager.cc:194] Registered new tserver with Master: e216be957abb467ab85f1919f063f407 (127.0.97.3:36927)
I20260620 18:29:15.678027 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53225
I20260620 18:29:15.682451 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:15.685938 629 heartbeater.cc:503] Master 127.0.97.62:44119 was elected leader, sending a full tablet report...
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:15.806942 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.784396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.0.97.1:43717)
0620 18:29:15.785324 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:15.785334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:15.785373 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:29:15.785435 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:15.785441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:15.785675 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:15.785809 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:15.787312 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.788417 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.789724 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.790292 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.790400 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.800446 (+ 10046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:15.800461 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:15.800476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:15.800522 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:15.803774 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.804476 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.804486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.804491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.804633 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.805031 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.805036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.805039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.805425 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:15.805852 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:15.806302 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 18:29:15.806589 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":573,"threads_started":1}
W20260620 18:29:15.808423 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34716
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:29:15 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:15.898800 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.883091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.0.97.1:43717)
0620 18:29:15.883317 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:15.883327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:15.883766 (+ 439us) server_negotiation.cc:407] Connection header received
0620 18:29:15.883904 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:15.883913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:15.884053 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:15.884184 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:15.885630 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.886996 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.888371 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.888898 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.889019 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.893253 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:15.893258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:15.893262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:15.893298 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:15.896139 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.896807 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.896811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.896813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.896930 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.897343 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.897346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.897348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.897607 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:15.897775 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:15.898193 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 18:29:15.898364 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:29:15.900705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 501
I20260620 18:29:15.915176 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:15.901683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.97.1:43717)
0620 18:29:15.901873 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:15.901881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:15.902206 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:29:15.902406 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:15.902411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:15.902482 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:15.902586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:15.904411 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.905165 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.906465 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:15.906894 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:15.907031 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:15.909545 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:15.909551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:15.909557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:15.909595 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:15.912313 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.912980 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.912987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.912991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.913183 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:15.913767 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:15.913773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:15.913776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:15.914062 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:15.914315 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:15.914623 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:29:15.914827 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:29:15.916020 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:16.002307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 501
I20260620 18:29:16.021621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 635
Jun 20 18:29:16 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:16.038080 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:16.022581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.97.2:32921)
0620 18:29:16.023163 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:16.023172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:16.023356 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:29:16.023592 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:16.023602 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:16.023881 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:16.024050 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:16.025256 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:16.026306 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:16.027667 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:16.028210 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:16.028356 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:16.032647 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:16.032659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:16.032674 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:16.032710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:16.035489 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:16.036188 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:16.036196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:16.036199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:16.036322 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:16.036709 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:16.036712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:16.036714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:16.037050 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:16.037263 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:16.037560 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:29:16.037754 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":315,"threads_started":1}
I20260620 18:29:16.039340 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:16.124361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 635
I20260620 18:29:16.149259 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 768
Jun 20 18:29:16 dist-test-slave-mcnm krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:16.165735 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:16.150161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.97.3:36927)
0620 18:29:16.150699 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:16.150707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:16.150736 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:29:16.150793 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:16.150800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:16.151051 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:16.151213 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:16.152665 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:16.153708 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:16.155076 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:16.155700 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:16.155807 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:16.159949 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:16.159984 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:16.160009 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:16.160041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:16.162975 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:16.163706 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:16.163713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:16.163715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:16.163833 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:16.164219 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:16.164224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:16.164226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:16.164572 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:16.164796 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:16.165099 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:29:16.165374 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":213,"threads_started":1}
I20260620 18:29:16.167016 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:16.252556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 768
I20260620 18:29:16.269320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 408
I20260620 18:29:16.270498 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:16.373030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 408
2026-06-20T18:29:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4082 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:18 dist-test-slave-mcnm krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:18 dist-test-slave-mcnm krb5kdc[917](info): set up 2 sockets
Jun 20 18:29:18 dist-test-slave-mcnm krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 18:29:19.250903 388 mini_kdc.cc:121] Time spent starting KDC: real 2.836s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 18:29:19.530731 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.182s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 18:29:19.711756 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.181s user 0.001s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/test-user.keytab.
W20260620 18:29:20.036037 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.324s user 0.001s sys 0.005s
Jun 20 18:29:20 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260620 18:29:21.142822 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 1.062s user 0.001s sys 0.007s
I20260620 18:29:21.143920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35643
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:21.477293 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:21.477568 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:21.477747 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:21.484592 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:21.484683 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:21.484766 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:21.484874 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:21.484941 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:21.494297 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:21.495606 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:21.496902 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:21.509101 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:21.508903 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:21.509302 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:21.510141 930 server_base.cc:1034] running on GCE node
I20260620 18:29:21.510929 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:21.512382 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:21.513607 930 hybrid_clock.cc:648] HybridClock initialized: now 1781980161513560 us; error 42 us; skew 500 ppm
I20260620 18:29:21.514001 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:21 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:21.516999 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:21.518386 930 webserver.cc:460] Webserver started at http://127.0.97.62:46651/ using document root <none> and password file <none>
I20260620 18:29:21.519006 930 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:21.519100 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:21.519331 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:21.522773 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "b696dda3ec7e477595d0db6c0d0d5404"
format_stamp: "Formatted at 2026-06-20 18:29:21 on dist-test-slave-mcnm"
server_key: "2c72f7ae8ed303693cd2fb9323a47a33"
server_key_iv: "ca988fe408386d489b6fcf231cb87ee5"
server_key_version: "encryptionkey@0"
I20260620 18:29:21.523437 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "b696dda3ec7e477595d0db6c0d0d5404"
format_stamp: "Formatted at 2026-06-20 18:29:21 on dist-test-slave-mcnm"
server_key: "2c72f7ae8ed303693cd2fb9323a47a33"
server_key_iv: "ca988fe408386d489b6fcf231cb87ee5"
server_key_version: "encryptionkey@0"
I20260620 18:29:21.528517 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:21.531912 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:21.533246 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:29:21.533433 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "b696dda3ec7e477595d0db6c0d0d5404"
format_stamp: "Formatted at 2026-06-20 18:29:21 on dist-test-slave-mcnm"
server_key: "2c72f7ae8ed303693cd2fb9323a47a33"
server_key_iv: "ca988fe408386d489b6fcf231cb87ee5"
server_key_version: "encryptionkey@0"
I20260620 18:29:21.533627 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:21.557157 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:21.561594 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:21.561937 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:21.582036 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35643
I20260620 18:29:21.582080 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35643 every 8 connection(s)
I20260620 18:29:21.583504 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:21.587568 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:21.593461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 930
I20260620 18:29:21.593700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:21.594524 388 external_mini_cluster.cc:1428] Setting key 0658dd84a4f9294316f8d1b9098e5019
I20260620 18:29:21.598614 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404: Bootstrap starting.
I20260620 18:29:21.602324 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:21.603365 997 log.cc:824] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:21.606228 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404: No bootstrap required, opened a new log
Jun 20 18:29:21 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:21.614513 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } }
I20260620 18:29:21.614929 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:21.615052 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b696dda3ec7e477595d0db6c0d0d5404, State: Initialized, Role: FOLLOWER
I20260620 18:29:21.615688 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } }
I20260620 18:29:21.615933 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:21.616079 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:21.616263 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:21.617774 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:21.596743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.0.97.62:35643)
0620 18:29:21.597528 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:21.597540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:21.597580 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:29:21.599284 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:21.599306 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:21.599758 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:21.600076 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:21.601867 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:21.603227 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:21.604785 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:21.605461 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:21.605595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:21.610173 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:21.610180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:21.610194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:21.610230 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:21.613419 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:21.614162 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:21.614171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:21.614179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:21.614361 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:21.614740 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:21.614745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:21.614749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:21.615479 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:21.615829 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:21.616355 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 18:29:21.616745 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":390,"threads_started":1}
I20260620 18:29:21.618656 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } }
I20260620 18:29:21.619189 997 leader_election.cc:302] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b696dda3ec7e477595d0db6c0d0d5404; no voters:
I20260620 18:29:21.619910 997 leader_election.cc:288] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:21.620028 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:21.621089 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [term 1 LEADER]: Becoming Leader. State: Replica: b696dda3ec7e477595d0db6c0d0d5404, State: Running, Role: LEADER
I20260620 18:29:21.621629 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } }
I20260620 18:29:21.622087 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:21.624120 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b696dda3ec7e477595d0db6c0d0d5404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } } }
I20260620 18:29:21.624477 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:21.624607 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b696dda3ec7e477595d0db6c0d0d5404. Latest consensus state: current_term: 1 leader_uuid: "b696dda3ec7e477595d0db6c0d0d5404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b696dda3ec7e477595d0db6c0d0d5404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35643 } } }
I20260620 18:29:21.624996 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:21.626921 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:21.631455 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:21.641079 1007 catalog_manager.cc:1347] Generated new cluster ID: 774daff4f5f344a58e3da4d55f994371
I20260620 18:29:21.641197 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:21.650682 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:21.652433 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:21.664112 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b696dda3ec7e477595d0db6c0d0d5404: Generated new TSK 0
I20260620 18:29:21.664974 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:21.713797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:22.033640 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:22.033905 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:22.034065 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:22.041461 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:22.041610 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:22.041828 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:22.052515 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:22.053807 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:22.055160 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:22.069205 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:22.069159 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:22.069263 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:22.072966 1023 server_base.cc:1034] running on GCE node
I20260620 18:29:22.073607 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:22.074563 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:22.075752 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781980162075705 us; error 49 us; skew 500 ppm
I20260620 18:29:22.076057 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:22 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:22.079267 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:22.080974 1023 webserver.cc:460] Webserver started at http://127.0.97.1:35237/ using document root <none> and password file <none>
I20260620 18:29:22.081753 1023 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:22.081908 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:22.082244 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:22.085337 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "49c07e8edbc344a48815b464abcad011"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "d4206eb30ee835af95b3f1aa850cb843"
server_key_iv: "688a3fcbebbb76a5535c6370669da40a"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.086191 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49c07e8edbc344a48815b464abcad011"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "d4206eb30ee835af95b3f1aa850cb843"
server_key_iv: "688a3fcbebbb76a5535c6370669da40a"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.091806 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:29:22.095331 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.096947 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:22.097151 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "49c07e8edbc344a48815b464abcad011"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "d4206eb30ee835af95b3f1aa850cb843"
server_key_iv: "688a3fcbebbb76a5535c6370669da40a"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.097388 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:22.143054 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:22.147974 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:22.148327 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:22.149339 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:22.150758 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:22.150878 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.151043 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:22.151129 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.186785 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34049
I20260620 18:29:22.186820 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34049 every 8 connection(s)
I20260620 18:29:22.188277 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:22.192373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1023
I20260620 18:29:22.192589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:22.193416 388 external_mini_cluster.cc:1428] Setting key fe0a449924c21f85bf99db80af269269
Jun 20 18:29:22 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:29:22.211047 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:22.190888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38293 (local address 127.0.97.62:35643)
0620 18:29:22.191351 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:22.191358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:22.192882 (+ 1524us) server_negotiation.cc:407] Connection header received
0620 18:29:22.195059 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:22.195064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:22.195147 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:22.195275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:22.197318 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.198020 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.200444 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.201210 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.201366 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:22.205237 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:22.205241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:22.205245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:22.205274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:22.207891 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:22.208807 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:22.208811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:22.208813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:22.208918 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:22.209390 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:22.209393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:22.209395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:22.209635 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:22.209739 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:22.210622 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 18:29:22.210769 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260620 18:29:22.212267 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:22.191637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35643 (local address 127.0.97.1:38293)
0620 18:29:22.192479 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:22.192583 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:22.194801 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:22.195575 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:22.195588 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:22.196310 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:22.197122 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:22.197151 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.198186 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.198194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:22.200291 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:22.200302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.201408 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.201413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:22.201652 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:22.202346 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:22.202367 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:22.205013 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:22.208031 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:22.208043 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:22.208064 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:22.208681 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:22.209025 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:22.209029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:22.209033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:22.209280 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:22.209760 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:22.209770 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:22.210214 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 18:29:22.210546 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 18:29:22.211099 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":311,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:29:22.213789 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35643
I20260620 18:29:22.214082 1151 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:22.214762 1151 heartbeater.cc:511] Master 127.0.97.62:35643 requested a full tablet report, sending...
I20260620 18:29:22.216784 962 ts_manager.cc:194] Registered new tserver with Master: 49c07e8edbc344a48815b464abcad011 (127.0.97.1:34049)
I20260620 18:29:22.218784 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:22.235565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:22.551800 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:22.552091 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:22.552187 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:22.559330 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:22.559443 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:22.559672 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:22.569614 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:22.570930 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:22.572337 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:22.587627 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:22.587792 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:22.588197 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:22.589272 1157 server_base.cc:1034] running on GCE node
I20260620 18:29:22.590106 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:22.591270 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:22.592554 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781980162592492 us; error 53 us; skew 500 ppm
I20260620 18:29:22.592932 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:22 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:22.596468 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:22.598284 1157 webserver.cc:460] Webserver started at http://127.0.97.2:38671/ using document root <none> and password file <none>
I20260620 18:29:22.599015 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:22.599174 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:22.599577 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:22.602957 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "c553ebf748be4826ab163c29351f9f4d"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "3c30a3dd14dd6fd66d3c057fc84e7671"
server_key_iv: "1d3794b17facf46e17bb38db822eac9c"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.603955 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c553ebf748be4826ab163c29351f9f4d"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "3c30a3dd14dd6fd66d3c057fc84e7671"
server_key_iv: "1d3794b17facf46e17bb38db822eac9c"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.609172 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:22.612690 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.614090 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:29:22.614260 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "c553ebf748be4826ab163c29351f9f4d"
format_stamp: "Formatted at 2026-06-20 18:29:22 on dist-test-slave-mcnm"
server_key: "3c30a3dd14dd6fd66d3c057fc84e7671"
server_key_iv: "1d3794b17facf46e17bb38db822eac9c"
server_key_version: "encryptionkey@0"
I20260620 18:29:22.614475 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:22.627643 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:22.632098 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:22.632428 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:22.633543 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:22.634910 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:22.635015 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.635164 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:22.635234 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:22.669061 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37699
I20260620 18:29:22.669107 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37699 every 8 connection(s)
I20260620 18:29:22.670363 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:22.671916 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1157
I20260620 18:29:22.672114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:22.672688 388 external_mini_cluster.cc:1428] Setting key 161a89f73ef745fc47162f55e2645c5b
Jun 20 18:29:22 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:29:22.691088 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:22.673194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39541 (local address 127.0.97.62:35643)
0620 18:29:22.673386 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:22.673391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:22.674844 (+ 1453us) server_negotiation.cc:407] Connection header received
0620 18:29:22.676666 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:22.676671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:22.676760 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:22.676877 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:22.678340 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.679025 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.680618 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.681037 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.681131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:22.685168 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:22.685173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:22.685176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:22.685206 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:22.688192 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:22.688932 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:22.688938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:22.688943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:22.689069 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:22.689492 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:22.689497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:22.689501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:22.689819 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:22.690019 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:22.690668 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:29:22.690837 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 18:29:22.692260 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:22.673808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35643 (local address 127.0.97.2:39541)
0620 18:29:22.674695 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:22.674733 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:22.676409 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:22.677103 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:22.677111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:22.677612 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:22.678147 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:22.678191 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.679180 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.679188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:22.680496 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:22.680507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:22.681255 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:22.681264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:22.681593 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:22.682312 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:22.682342 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:22.684942 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:22.688336 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:22.688347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:22.688363 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:22.688790 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:22.689181 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:22.689187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:22.689191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:22.689373 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:22.689958 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:22.689965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:22.690315 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 18:29:22.690586 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:29:22.690993 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":297,"threads_started":1}
I20260620 18:29:22.693782 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35643
I20260620 18:29:22.694090 1285 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:22.694772 1285 heartbeater.cc:511] Master 127.0.97.62:35643 requested a full tablet report, sending...
I20260620 18:29:22.696259 962 ts_manager.cc:194] Registered new tserver with Master: c553ebf748be4826ab163c29351f9f4d (127.0.97.2:37699)
I20260620 18:29:22.697520 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:22.715293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:23.048157 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:23.048460 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:23.048558 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:23.055773 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:23.055873 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:23.055985 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:23.065521 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:23.066692 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:23.068162 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:23.080911 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:23.080958 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:23.080871 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:23.082036 1290 server_base.cc:1034] running on GCE node
I20260620 18:29:23.082664 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:23.083722 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:23.084954 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781980163084911 us; error 51 us; skew 500 ppm
I20260620 18:29:23.085274 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:23.088476 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:23.089865 1290 webserver.cc:460] Webserver started at http://127.0.97.3:43013/ using document root <none> and password file <none>
I20260620 18:29:23.090531 1290 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:23.090673 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:23.090991 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:23.093855 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "25eb5110bbf84f40adcbc9ee17a36d3b"
format_stamp: "Formatted at 2026-06-20 18:29:23 on dist-test-slave-mcnm"
server_key: "cfff8ac031150cf4f01eecab97584726"
server_key_iv: "796eb6fb1190cb958d2f1b42419cb637"
server_key_version: "encryptionkey@0"
I20260620 18:29:23.094611 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "25eb5110bbf84f40adcbc9ee17a36d3b"
format_stamp: "Formatted at 2026-06-20 18:29:23 on dist-test-slave-mcnm"
server_key: "cfff8ac031150cf4f01eecab97584726"
server_key_iv: "796eb6fb1190cb958d2f1b42419cb637"
server_key_version: "encryptionkey@0"
I20260620 18:29:23.099414 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:29:23.102883 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:23.104259 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:23.104449 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "25eb5110bbf84f40adcbc9ee17a36d3b"
format_stamp: "Formatted at 2026-06-20 18:29:23 on dist-test-slave-mcnm"
server_key: "cfff8ac031150cf4f01eecab97584726"
server_key_iv: "796eb6fb1190cb958d2f1b42419cb637"
server_key_version: "encryptionkey@0"
I20260620 18:29:23.104652 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:23.120823 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:23.125393 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:23.125696 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:23.126667 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:23.128126 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:23.128232 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:23.128387 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:23.128465 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:23.163650 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36417
I20260620 18:29:23.163688 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36417 every 8 connection(s)
I20260620 18:29:23.164938 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:23.172789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1290
I20260620 18:29:23.172999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:23.173411 388 external_mini_cluster.cc:1428] Setting key e5d5a0ea1b3f26deda34c681bd726d0c
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:23.186707 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.167450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58667 (local address 127.0.97.62:35643)
0620 18:29:23.167650 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.167654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.169002 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 18:29:23.170837 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.170843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.170937 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.171049 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.172661 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.173591 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.176860 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.177326 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.177428 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.180964 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.180969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.180972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.181004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.183507 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.184274 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.184278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.184280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.184412 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.185028 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.185032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.185034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.185290 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.185421 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.186154 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.186365 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:29:23.187829 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.168063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35643 (local address 127.0.97.3:58667)
0620 18:29:23.168814 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.168861 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.170559 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.171284 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.171292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.171869 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:23.172444 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.172469 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.173760 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.173766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.176065 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.176079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.177520 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.177525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.177718 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.178359 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:23.178380 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:23.180706 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:23.183699 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:23.183706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:23.183718 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:23.184133 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:23.184675 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:23.184679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:23.184681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:23.184876 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:23.185435 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:23.185441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:23.185748 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.185974 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.186534 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
I20260620 18:29:23.189296 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35643
I20260620 18:29:23.189607 1418 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:23.190454 1418 heartbeater.cc:511] Master 127.0.97.62:35643 requested a full tablet report, sending...
I20260620 18:29:23.191965 962 ts_manager.cc:194] Registered new tserver with Master: 25eb5110bbf84f40adcbc9ee17a36d3b (127.0.97.3:36417)
I20260620 18:29:23.192754 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58667
I20260620 18:29:23.202126 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:23.220491 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.208276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.0.97.62:35643)
0620 18:29:23.208439 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.208444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.208931 (+ 487us) server_negotiation.cc:407] Connection header received
0620 18:29:23.209130 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.209133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.209192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.209282 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.211073 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.211862 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.213026 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.213462 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.213559 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.215727 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.215732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.215735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.215765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.218160 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.218746 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.218750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.218752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.218882 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.219188 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.219191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.219193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.219471 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.219725 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.219995 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.220149 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:29:23.221468 1151 heartbeater.cc:503] Master 127.0.97.62:35643 was elected leader, sending a full tablet report...
I20260620 18:29:23.226483 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:23.233431 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:23.272061 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.245672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.0.97.2:37699)
0620 18:29:23.260851 (+ 15179us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.260860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.260889 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:29:23.260980 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.260987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.261733 (+ 746us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.263356 (+ 1623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:23.265067 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.266652 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.269807 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.270948 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.271096 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.271294 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.271574 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.271734 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15005,"thread_start_us":17396,"threads_started":1}
I20260620 18:29:23.272132 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.244963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37699 (local address 127.0.0.1:39062)
0620 18:29:23.260058 (+ 15095us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.260135 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.260380 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.264447 (+ 4067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.264459 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.264497 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:23.264888 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.264900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.266839 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.266847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.269648 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.269661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.271140 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.271146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.271316 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.271374 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.271496 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.271710 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14922,"mutex_wait_us":1,"spinlock_wait_cycles":33024,"thread_start_us":15642,"threads_started":1}
I20260620 18:29:23.273099 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.245004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.0.97.3:36417)
0620 18:29:23.264220 (+ 19216us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.264233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.264273 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:29:23.264386 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.264398 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.264695 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.264930 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:23.265779 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.267382 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.270036 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.271788 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.271926 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.272142 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.272627 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.272789 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19001,"thread_start_us":18791,"threads_started":1}
I20260620 18:29:23.273126 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.244691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36417 (local address 127.0.0.1:55422)
0620 18:29:23.260050 (+ 15359us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.260135 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.260381 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.265240 (+ 4859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.265245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.265267 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:23.265598 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.265607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.267568 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.267574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.269892 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.269902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.272270 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.272276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.272423 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.272455 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.272554 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.272690 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14948,"spinlock_wait_cycles":3840,"thread_start_us":15913,"threads_started":1}
I20260620 18:29:23.273664 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.261784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34049 (local address 127.0.0.1:38868)
0620 18:29:23.264465 (+ 2681us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.264489 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.264605 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.266060 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.266064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.266091 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:23.266398 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.266409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.268084 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.268092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.270197 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.270211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.272632 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.272638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.272786 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.272824 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.272906 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.273021 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2541,"mutex_wait_us":2,"thread_start_us":2406,"threads_started":1}
I20260620 18:29:23.274286 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.261875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.0.97.1:34049)
0620 18:29:23.265559 (+ 3684us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.265567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.265593 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:29:23.265671 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.265678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.265870 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.266049 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:23.266572 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.267884 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.270355 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.272166 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.272285 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.272503 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.273209 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.273975 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3523,"thread_start_us":3329,"threads_started":1}
I20260620 18:29:23.276625 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 48c5accfe98c4c04a9afb25fbcf9c967 (DEFAULT_TABLE table=test-table [id=a649a52ea52a4a5692af2da176ea4135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:23.276688 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 48c5accfe98c4c04a9afb25fbcf9c967 (DEFAULT_TABLE table=test-table [id=a649a52ea52a4a5692af2da176ea4135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:23.277868 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c5accfe98c4c04a9afb25fbcf9c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:23.277482 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 48c5accfe98c4c04a9afb25fbcf9c967 (DEFAULT_TABLE table=test-table [id=a649a52ea52a4a5692af2da176ea4135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:23.278045 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c5accfe98c4c04a9afb25fbcf9c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:23.278661 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c5accfe98c4c04a9afb25fbcf9c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:23.288681 1436 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Bootstrap starting.
I20260620 18:29:23.291864 1436 tablet_bootstrap.cc:654] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:23.292457 1437 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Bootstrap starting.
I20260620 18:29:23.293208 1438 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Bootstrap starting.
I20260620 18:29:23.293229 1436 log.cc:824] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:23.296411 1437 tablet_bootstrap.cc:654] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:23.297446 1437 log.cc:824] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:23.298015 1438 tablet_bootstrap.cc:654] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:23.299158 1438 log.cc:824] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:23.300407 1437 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: No bootstrap required, opened a new log
I20260620 18:29:23.300546 1436 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: No bootstrap required, opened a new log
I20260620 18:29:23.300678 1437 ts_tablet_manager.cc:1397] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 18:29:23.300840 1436 ts_tablet_manager.cc:1397] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 18:29:23.301946 1438 tablet_bootstrap.cc:492] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: No bootstrap required, opened a new log
I20260620 18:29:23.302187 1438 ts_tablet_manager.cc:1397] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 18:29:23.308637 1436 raft_consensus.cc:348] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.309122 1436 raft_consensus.cc:374] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:23.309262 1436 raft_consensus.cc:729] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553ebf748be4826ab163c29351f9f4d, State: Initialized, Role: FOLLOWER
I20260620 18:29:23.309870 1436 consensus_queue.cc:260] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.310950 1285 heartbeater.cc:503] Master 127.0.97.62:35643 was elected leader, sending a full tablet report...
I20260620 18:29:23.311872 1437 raft_consensus.cc:348] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.312259 1437 raft_consensus.cc:374] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:23.312340 1437 raft_consensus.cc:729] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c07e8edbc344a48815b464abcad011, State: Initialized, Role: FOLLOWER
I20260620 18:29:23.312819 1436 ts_tablet_manager.cc:1428] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 18:29:23.312949 1437 consensus_queue.cc:260] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.313621 1438 raft_consensus.cc:348] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.314093 1438 raft_consensus.cc:374] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:23.314198 1438 raft_consensus.cc:729] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb5110bbf84f40adcbc9ee17a36d3b, State: Initialized, Role: FOLLOWER
I20260620 18:29:23.314940 1438 consensus_queue.cc:260] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.320120 1418 heartbeater.cc:503] Master 127.0.97.62:35643 was elected leader, sending a full tablet report...
I20260620 18:29:23.321026 1437 ts_tablet_manager.cc:1428] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Time spent starting tablet: real 0.020s user 0.003s sys 0.016s
I20260620 18:29:23.322103 1438 ts_tablet_manager.cc:1428] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
I20260620 18:29:23.386951 1442 raft_consensus.cc:482] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:23.387290 1442 raft_consensus.cc:504] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.389043 1442 leader_election.cc:288] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb5110bbf84f40adcbc9ee17a36d3b (127.0.97.3:36417), 49c07e8edbc344a48815b464abcad011 (127.0.97.1:34049)
I20260620 18:29:23.398720 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.389520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41569 (local address 127.0.97.3:36417)
0620 18:29:23.389769 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.389779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.390690 (+ 911us) server_negotiation.cc:407] Connection header received
0620 18:29:23.390977 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.390987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.391071 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.391145 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:23.392077 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.393212 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.396517 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.397732 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.397863 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.397965 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.398309 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.398447 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:29:23.398836 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.389431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36417 (local address 127.0.97.2:41569)
0620 18:29:23.390549 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.390579 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.390754 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.391379 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.391386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.391414 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:23.391909 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.391923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.393360 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.393364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.396347 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.396356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.397958 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.397965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.398107 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.398138 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.398284 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.398425 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":172,"thread_start_us":367,"threads_started":1}
I20260620 18:29:23.398928 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.390777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35857 (local address 127.0.97.1:34049)
0620 18:29:23.391019 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.391027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.391282 (+ 255us) server_negotiation.cc:407] Connection header received
0620 18:29:23.391455 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.391458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.391565 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.391645 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:23.392342 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.393382 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.396511 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.397721 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.397870 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.397965 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.398520 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.398640 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 18:29:23.399650 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.390541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34049 (local address 127.0.97.2:35857)
0620 18:29:23.391181 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:23.391199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:23.391326 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:23.391691 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:23.391696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:23.391723 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:23.392184 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.392193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.393511 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.393519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.396338 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:23.396355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.398137 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.398143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:23.398333 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.398364 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:29:23.399105 (+ 741us) client_negotiation.cc:240] Negotiation successful
0620 18:29:23.399237 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":45,"thread_start_us":341,"threads_started":1}
I20260620 18:29:23.400166 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c5accfe98c4c04a9afb25fbcf9c967" candidate_uuid: "c553ebf748be4826ab163c29351f9f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" is_pre_election: true
I20260620 18:29:23.400687 1373 raft_consensus.cc:2454] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c553ebf748be4826ab163c29351f9f4d in term 0.
I20260620 18:29:23.400667 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c5accfe98c4c04a9afb25fbcf9c967" candidate_uuid: "c553ebf748be4826ab163c29351f9f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c07e8edbc344a48815b464abcad011" is_pre_election: true
I20260620 18:29:23.401273 1106 raft_consensus.cc:2454] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c553ebf748be4826ab163c29351f9f4d in term 0.
I20260620 18:29:23.401412 1174 leader_election.cc:302] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb5110bbf84f40adcbc9ee17a36d3b, c553ebf748be4826ab163c29351f9f4d; no voters:
I20260620 18:29:23.401738 1442 raft_consensus.cc:2781] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:23.401887 1442 raft_consensus.cc:482] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:23.401993 1442 raft_consensus.cc:3037] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:23.404989 1442 raft_consensus.cc:504] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.405686 1442 leader_election.cc:288] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [CANDIDATE]: Term 1 election: Requested vote from peers 25eb5110bbf84f40adcbc9ee17a36d3b (127.0.97.3:36417), 49c07e8edbc344a48815b464abcad011 (127.0.97.1:34049)
I20260620 18:29:23.406103 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c5accfe98c4c04a9afb25fbcf9c967" candidate_uuid: "c553ebf748be4826ab163c29351f9f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b"
I20260620 18:29:23.406172 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c5accfe98c4c04a9afb25fbcf9c967" candidate_uuid: "c553ebf748be4826ab163c29351f9f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c07e8edbc344a48815b464abcad011"
I20260620 18:29:23.406266 1373 raft_consensus.cc:3037] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:23.406399 1106 raft_consensus.cc:3037] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:23.408604 1373 raft_consensus.cc:2454] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c553ebf748be4826ab163c29351f9f4d in term 1.
I20260620 18:29:23.408968 1174 leader_election.cc:302] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb5110bbf84f40adcbc9ee17a36d3b, c553ebf748be4826ab163c29351f9f4d; no voters:
I20260620 18:29:23.409104 1106 raft_consensus.cc:2454] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c553ebf748be4826ab163c29351f9f4d in term 1.
I20260620 18:29:23.409248 1442 raft_consensus.cc:2781] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:23.409742 1442 raft_consensus.cc:686] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [term 1 LEADER]: Becoming Leader. State: Replica: c553ebf748be4826ab163c29351f9f4d, State: Running, Role: LEADER
I20260620 18:29:23.410202 1442 consensus_queue.cc:237] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } }
I20260620 18:29:23.414598 961 catalog_manager.cc:5495] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to c553ebf748be4826ab163c29351f9f4d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c553ebf748be4826ab163c29351f9f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb5110bbf84f40adcbc9ee17a36d3b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c553ebf748be4826ab163c29351f9f4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c07e8edbc344a48815b464abcad011" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34049 } health_report { overall_health: UNKNOWN } } }
W20260620 18:29:23.416440 1419 tablet.cc:2369] T 48c5accfe98c4c04a9afb25fbcf9c967 P 25eb5110bbf84f40adcbc9ee17a36d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:23.422267 1286 tablet.cc:2369] T 48c5accfe98c4c04a9afb25fbcf9c967 P c553ebf748be4826ab163c29351f9f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:23.440945 1152 tablet.cc:2369] T 48c5accfe98c4c04a9afb25fbcf9c967 P 49c07e8edbc344a48815b464abcad011: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:23.453624 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.438386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.0.97.1:34049)
0620 18:29:23.438642 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.438650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.439048 (+ 398us) server_negotiation.cc:407] Connection header received
0620 18:29:23.439257 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.439262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.439358 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.439468 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.440980 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.441817 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.443059 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.443487 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.443652 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.447868 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.447883 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.447900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.447944 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.451065 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.451839 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.451846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.451849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.451981 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.452311 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.452314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.452316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.452660 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.452892 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.453189 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.453359 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:23.498848 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.485313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.97.1:34049)
0620 18:29:23.485550 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.485556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.485721 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:29:23.485898 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.485901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.485964 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.486047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.487376 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.488168 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.489600 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.490016 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.490111 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.493940 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.493945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.493949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.493982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.496613 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.497211 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.497215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.497218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.497355 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.497685 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.497688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.497690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.497937 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.498134 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.498324 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.498476 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:23.544359 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.530347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.97.1:34049)
0620 18:29:23.530577 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.530586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.530914 (+ 328us) server_negotiation.cc:407] Connection header received
0620 18:29:23.531130 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.531134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.531196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.531305 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.533064 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.533745 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.535002 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.535421 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.535586 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.539577 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.539583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.539587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.539620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.542106 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.542750 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.542754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.542757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.542866 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.543178 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.543182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.543184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.543465 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.543757 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.543948 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.544113 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:29:23.546068 1066 tablet_service.cc:2908] Rejecting scan request for tablet 48c5accfe98c4c04a9afb25fbcf9c967: Uninitialized: clean time has not yet been initialized
I20260620 18:29:23.549055 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1023
I20260620 18:29:23.561546 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.549944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.0.97.1:34049)
0620 18:29:23.550127 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.550132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.550500 (+ 368us) server_negotiation.cc:407] Connection header received
0620 18:29:23.550599 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.550602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.550684 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.550790 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.551979 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.552680 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.553825 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.554203 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.554350 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.556475 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.556480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.556483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.556513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.558997 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.559665 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.559670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.559672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.559798 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.560162 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.560165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.560167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.560466 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.561073 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.561172 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.561326 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:29:23.562235 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:23.656422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1023
I20260620 18:29:23.673724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1157
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:23.688616 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.674687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.0.97.2:37699)
0620 18:29:23.674899 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.674907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.674930 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:29:23.674999 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.675003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.675059 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.675169 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.676399 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.677257 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.678541 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.678979 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.679112 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.683113 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.683123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.683136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.683165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.686055 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.686730 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.686736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.686739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.686857 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.687217 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.687222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.687225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.687642 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.687898 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.688102 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.688278 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:29:23.689266 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:23.778131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1157
I20260620 18:29:23.794529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1290
Jun 20 18:29:23 dist-test-slave-mcnm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:23.809252 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:23.795304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.97.3:36417)
0620 18:29:23.795454 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:23.795459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:23.795477 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:29:23.795648 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:23.795651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:23.795711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:23.795802 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:23.797024 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.797729 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.799001 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:23.799367 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:23.799460 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:23.803720 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:23.803742 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:23.803759 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:23.803796 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:23.806336 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.807041 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.807050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.807052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.807180 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:23.807720 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:23.807726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:23.807729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:23.808063 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:23.808367 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:23.808726 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 18:29:23.808941 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:29:23.809921 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:23.897478 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1290
I20260620 18:29:23.915593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 930
I20260620 18:29:23.916848 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:24.024521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 930
2026-06-20T18:29:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7653 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:24 dist-test-slave-mcnm krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:24 dist-test-slave-mcnm krb5kdc[1466](info): set up 2 sockets
Jun 20 18:29:24 dist-test-slave-mcnm krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 18:29:25.111573 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:25 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:25.291474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39939
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:25.603892 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:25.604177 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:25.604339 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:25.611487 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:25.611649 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:29:25.611720 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:25.611828 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:25.611918 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:25.612006 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:25.621645 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:25.622936 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:25.624449 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:25.636044 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:25.636086 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:25.636343 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:25.637135 1479 server_base.cc:1034] running on GCE node
I20260620 18:29:25.637943 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:25.639314 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:25.640566 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781980165640503 us; error 39 us; skew 500 ppm
I20260620 18:29:25.641142 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:25 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:25.644399 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:25.646072 1479 webserver.cc:460] Webserver started at http://127.0.97.62:40291/ using document root <none> and password file <none>
I20260620 18:29:25.646826 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:25.646965 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:25.647312 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:25.650460 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a"
format_stamp: "Formatted at 2026-06-20 18:29:25 on dist-test-slave-mcnm"
server_key: "35f7ce81cf451cbff0c3c894de757f71"
server_key_iv: "b619da04d24e869fe6915867ed8540a5"
server_key_version: "encryptionkey@0"
I20260620 18:29:25.651319 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a"
format_stamp: "Formatted at 2026-06-20 18:29:25 on dist-test-slave-mcnm"
server_key: "35f7ce81cf451cbff0c3c894de757f71"
server_key_iv: "b619da04d24e869fe6915867ed8540a5"
server_key_version: "encryptionkey@0"
I20260620 18:29:25.656862 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:29:25.660339 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:25.661624 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:25.661803 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a"
format_stamp: "Formatted at 2026-06-20 18:29:25 on dist-test-slave-mcnm"
server_key: "35f7ce81cf451cbff0c3c894de757f71"
server_key_iv: "b619da04d24e869fe6915867ed8540a5"
server_key_version: "encryptionkey@0"
I20260620 18:29:25.662030 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:25.683919 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:25.688495 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:25.688781 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:25.711045 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39939
I20260620 18:29:25.711063 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39939 every 8 connection(s)
I20260620 18:29:25.712481 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:25.716270 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:25.717587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1479
I20260620 18:29:25.717826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:25.718621 388 external_mini_cluster.cc:1428] Setting key 1fdde4abe56f3695dae9e2bef45f555b
I20260620 18:29:25.731194 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Bootstrap starting.
Jun 20 18:29:25 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:25.734944 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:25.736193 1556 log.cc:824] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:25.738983 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: No bootstrap required, opened a new log
I20260620 18:29:25.742403 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:25.720835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.97.62:39939)
0620 18:29:25.721670 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:25.721686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:25.721742 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:29:25.723340 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:25.723362 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:25.723858 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:25.724210 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:25.725941 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:25.727287 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:25.729126 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:25.729808 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:25.729947 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:25.734687 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:25.734695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:25.734711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:25.734750 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:25.737872 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:25.738549 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:25.738555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:25.738562 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:25.738738 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:25.739142 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:25.739147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:25.739150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:25.739883 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:25.740176 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:25.740549 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:29:25.741029 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":419,"threads_started":1}
I20260620 18:29:25.747865 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } }
I20260620 18:29:25.748189 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:25.748262 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae5b6985ccc4c4ab06b8c19f524ae9a, State: Initialized, Role: FOLLOWER
I20260620 18:29:25.748718 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } }
I20260620 18:29:25.748909 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:25.749004 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:25.749132 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:25.751351 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } }
I20260620 18:29:25.751863 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae5b6985ccc4c4ab06b8c19f524ae9a; no voters:
I20260620 18:29:25.752435 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:25.752533 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:25.753374 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [term 1 LEADER]: Becoming Leader. State: Replica: 8ae5b6985ccc4c4ab06b8c19f524ae9a, State: Running, Role: LEADER
I20260620 18:29:25.753880 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } }
I20260620 18:29:25.754521 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:25.756738 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae5b6985ccc4c4ab06b8c19f524ae9a. Latest consensus state: current_term: 1 leader_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } } }
I20260620 18:29:25.757145 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:25.756784 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae5b6985ccc4c4ab06b8c19f524ae9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39939 } } }
I20260620 18:29:25.758311 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:25.759953 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:25.765718 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:25.774776 1568 catalog_manager.cc:1347] Generated new cluster ID: 0e76995930844cb79ba2f666eeeda645
I20260620 18:29:25.774907 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:25.785007 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:25.786262 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:25.796608 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Generated new TSK 0
I20260620 18:29:25.797501 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:25.857216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39939
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:26.193408 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:26.193734 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:26.193898 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:26.201190 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:26.201347 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:26.201489 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:26.201622 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:26.211390 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:26.212709 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:26.214046 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:26.227571 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:26.227608 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:26.227563 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:26.228459 1582 server_base.cc:1034] running on GCE node
I20260620 18:29:26.229076 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:26.230083 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:26.231340 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781980166231249 us; error 62 us; skew 500 ppm
I20260620 18:29:26.231683 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:26 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:26.234747 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:26.236316 1582 webserver.cc:460] Webserver started at http://127.0.97.1:36001/ using document root <none> and password file <none>
I20260620 18:29:26.236961 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:26.237048 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:26.237318 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:26.240221 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "967dddd973c542039f40bc27b0277399"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "9be641aaf0abc5573dd79da234f10f07"
server_key_iv: "230c4012e006510d32e84533379dbd13"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.241046 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "967dddd973c542039f40bc27b0277399"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "9be641aaf0abc5573dd79da234f10f07"
server_key_iv: "230c4012e006510d32e84533379dbd13"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.246388 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:29:26.249974 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:26.251654 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:29:26.251785 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "967dddd973c542039f40bc27b0277399"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "9be641aaf0abc5573dd79da234f10f07"
server_key_iv: "230c4012e006510d32e84533379dbd13"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.251971 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:26.278093 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:26.282275 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:26.282500 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:26.285108 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:26.285199 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:26.285348 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:26.285409 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:29:26 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:26.310182 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.285728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41419 (local address 127.0.97.62:39939)
0620 18:29:26.286287 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:26.286294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:26.287358 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 18:29:26.289424 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:26.289433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:26.289541 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:26.289666 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:26.292179 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.293525 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.296010 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.297010 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.297151 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.302341 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:26.302351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:26.302359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:26.302415 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:26.306282 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.307447 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.307454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.307458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.307643 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.308301 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.308307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.308311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.308627 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:26.308799 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:26.309547 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 18:29:26.309769 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":335,"threads_started":1}
I20260620 18:29:26.311261 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.286290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.1:41419)
0620 18:29:26.287179 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:26.287226 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:26.289074 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:26.289978 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:26.289992 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:26.290825 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:26.291893 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.291916 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.293727 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.293734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.295832 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.295843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.297311 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.297319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.297529 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.298354 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:26.298398 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:26.302036 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:26.306457 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.306468 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.306491 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.307281 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.307798 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.307809 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.307813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.308157 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.308771 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:26.308779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:26.309158 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 18:29:26.309437 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 18:29:26.309981 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":396,"threads_started":1}
I20260620 18:29:26.334762 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.317773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46979 (local address 127.0.97.62:39939)
0620 18:29:26.318023 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:26.318033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:26.318667 (+ 634us) server_negotiation.cc:407] Connection header received
0620 18:29:26.318879 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:26.318884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:26.319004 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:26.319097 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:26.320902 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.322107 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.324878 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.325615 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.325753 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.328586 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:26.328593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:26.328599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:26.328651 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:26.332109 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.332786 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.332792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.332798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.332969 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.333356 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.333361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.333364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.333633 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:26.333782 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:26.334185 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 18:29:26.334399 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:29:26.334864 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.317757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.1:46979)
0620 18:29:26.318524 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:26.318547 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:26.318731 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:26.319157 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:26.319166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:26.320024 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:26.320684 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.320707 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.322325 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.322334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.324694 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.324710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.326135 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.326145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.326400 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.327283 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:26.327301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:26.328390 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:26.332253 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.332260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.332264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.332658 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.333108 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.333111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.333118 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.333233 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.333748 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:26.333752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:26.333929 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 18:29:26.334066 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:29:26.334393 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":410,"threads_started":1}
I20260620 18:29:26.343127 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46883
I20260620 18:29:26.343173 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46883 every 8 connection(s)
I20260620 18:29:26.344408 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:26.346474 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39939
I20260620 18:29:26.346711 1718 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:26.347322 1718 heartbeater.cc:511] Master 127.0.97.62:39939 requested a full tablet report, sending...
I20260620 18:29:26.348223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1582
I20260620 18:29:26.348476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:26.349085 388 external_mini_cluster.cc:1428] Setting key b1cc6b80da81ef7d17fdb7881edb252d
I20260620 18:29:26.349403 1511 ts_manager.cc:194] Registered new tserver with Master: 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883)
I20260620 18:29:26.351624 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41419
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:26.391207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39939
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:26.706813 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:26.707085 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:26.707180 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:26.714502 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:26.714665 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:26.714828 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:26.714953 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:26.724836 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:26.726179 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:26.727571 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:26.741138 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:26.741102 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:26.741165 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:26.742271 1722 server_base.cc:1034] running on GCE node
I20260620 18:29:26.742878 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:26.743870 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:26.745113 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781980166745060 us; error 43 us; skew 500 ppm
I20260620 18:29:26.745411 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:26 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:26.752236 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:26.753628 1722 webserver.cc:460] Webserver started at http://127.0.97.2:42197/ using document root <none> and password file <none>
I20260620 18:29:26.754416 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:26.754552 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:26.754880 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:26.758143 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "0647823b840f44d9aa08117b0397a1ae"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "8a2790eaed7c0e6ada779895b1438d7b"
server_key_iv: "4274b96b024d733b2eebb153b2c84948"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.758939 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0647823b840f44d9aa08117b0397a1ae"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "8a2790eaed7c0e6ada779895b1438d7b"
server_key_iv: "4274b96b024d733b2eebb153b2c84948"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.766229 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260620 18:29:26.770952 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:26.772372 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:26.772542 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "0647823b840f44d9aa08117b0397a1ae"
format_stamp: "Formatted at 2026-06-20 18:29:26 on dist-test-slave-mcnm"
server_key: "8a2790eaed7c0e6ada779895b1438d7b"
server_key_iv: "4274b96b024d733b2eebb153b2c84948"
server_key_version: "encryptionkey@0"
I20260620 18:29:26.772758 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:26.794077 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:26.798868 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:26.799187 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:26.801975 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:26.802100 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:26.802268 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:26.802362 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:29:26 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:26.826339 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.802656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50979 (local address 127.0.97.62:39939)
0620 18:29:26.802861 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:26.802868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:26.804388 (+ 1520us) server_negotiation.cc:407] Connection header received
0620 18:29:26.806648 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:26.806659 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:26.806779 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:26.806913 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:26.809035 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.810485 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.812967 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.813466 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.813596 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.818207 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:26.818217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:26.818227 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:26.818320 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:26.822307 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.823271 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.823278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.823281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.823409 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.824039 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.824044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.824047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.824391 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:26.824558 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:26.825723 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0620 18:29:26.826034 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:29:26.828558 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.803227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.2:50979)
0620 18:29:26.804190 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:26.804243 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:26.806246 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:26.807176 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:26.807196 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:26.807990 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:26.808811 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.808845 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.810666 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.810674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.812798 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.812810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.813719 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.813725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.814047 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.814889 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:26.814945 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:26.817949 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:26.822458 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.822469 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.822487 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.823129 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.823545 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.823550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.823556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.823886 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.824567 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:26.824579 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:26.825184 (+ 605us) client_negotiation.cc:769] Sending connection context
0620 18:29:26.825648 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 18:29:26.826422 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":478,"threads_started":1}
I20260620 18:29:26.854328 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.836357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.2:38981)
0620 18:29:26.840591 (+ 4234us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:26.840620 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:26.840813 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:26.841210 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:26.841215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:26.841976 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:26.842530 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.842547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.843563 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.843570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.845027 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:26.845043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.846275 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.846283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:26.846446 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.847155 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:26.847187 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:26.848387 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:26.851296 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.851304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.851308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.851883 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.852283 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:26.852291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:26.852299 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:26.852508 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:26.853188 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:26.853193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:26.853409 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 18:29:26.853616 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 18:29:26.854008 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3995,"thread_start_us":3804,"threads_started":1}
I20260620 18:29:26.854344 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:26.836436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38981 (local address 127.0.97.62:39939)
0620 18:29:26.836692 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:26.836701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:26.840728 (+ 4027us) server_negotiation.cc:407] Connection header received
0620 18:29:26.840972 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:26.840979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:26.841073 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:26.841186 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:26.842673 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.843393 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.845230 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:26.845824 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:26.845987 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:26.848563 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:26.848569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:26.848575 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:26.848609 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:26.851167 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.852018 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.852022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.852024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.852132 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:26.852672 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:26.852679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:26.852683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:26.853072 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:26.853231 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:26.853799 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 18:29:26.854038 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:29:26.856190 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43903 every 8 connection(s)
I20260620 18:29:26.856288 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43903
I20260620 18:29:26.857918 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:26.858860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1722
I20260620 18:29:26.859093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:26.859687 388 external_mini_cluster.cc:1428] Setting key a00dbac0c7562440f05db2bf9b69a751
I20260620 18:29:26.860764 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39939
I20260620 18:29:26.861052 1857 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:26.861783 1857 heartbeater.cc:511] Master 127.0.97.62:39939 requested a full tablet report, sending...
I20260620 18:29:26.863425 1511 ts_manager.cc:194] Registered new tserver with Master: 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:26.864717 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50979
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:26.903939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39939
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:27.229905 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:27.230242 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:27.230348 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:27.238188 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:27.238286 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:27.238368 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:27.238447 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:27.248725 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:27.250037 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:27.251425 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:27.265059 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:27.265085 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:27.265085 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:27.265831 1861 server_base.cc:1034] running on GCE node
I20260620 18:29:27.266505 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:27.267416 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:27.268630 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781980167268569 us; error 34 us; skew 500 ppm
I20260620 18:29:27.268927 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:27 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:27.272080 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:27.273782 1861 webserver.cc:460] Webserver started at http://127.0.97.3:38107/ using document root <none> and password file <none>
I20260620 18:29:27.274482 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:27.274570 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:27.274833 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:27.277838 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "185f50f120244da3b46afd0b6fab1f6d"
format_stamp: "Formatted at 2026-06-20 18:29:27 on dist-test-slave-mcnm"
server_key: "0150e35e1e60a667bab768f2c4b9727e"
server_key_iv: "138145f8ced01102a6444e4c584b180d"
server_key_version: "encryptionkey@0"
I20260620 18:29:27.278564 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "185f50f120244da3b46afd0b6fab1f6d"
format_stamp: "Formatted at 2026-06-20 18:29:27 on dist-test-slave-mcnm"
server_key: "0150e35e1e60a667bab768f2c4b9727e"
server_key_iv: "138145f8ced01102a6444e4c584b180d"
server_key_version: "encryptionkey@0"
I20260620 18:29:27.283396 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:29:27.286759 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:27.288261 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:27.288430 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "185f50f120244da3b46afd0b6fab1f6d"
format_stamp: "Formatted at 2026-06-20 18:29:27 on dist-test-slave-mcnm"
server_key: "0150e35e1e60a667bab768f2c4b9727e"
server_key_iv: "138145f8ced01102a6444e4c584b180d"
server_key_version: "encryptionkey@0"
I20260620 18:29:27.288648 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:27.313970 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:27.318305 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:27.318610 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:27.321328 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:27.321434 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:27.321581 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:27.321686 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 18:29:27 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:27.343745 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.321790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40849 (local address 127.0.97.62:39939)
0620 18:29:27.322043 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.322052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.323485 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 18:29:27.325685 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.325690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.325780 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.325890 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:27.328275 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.329454 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.331664 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.332327 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.332501 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.337353 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:27.337358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:27.337361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:27.337392 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:27.340266 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.341288 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.341295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.341299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.341473 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.341999 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.342004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.342008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.342299 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:27.342498 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.343162 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.343372 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:29:27.344848 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.322485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.3:40849)
0620 18:29:27.323330 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.323368 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.325402 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.326136 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.326144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.326812 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:27.328054 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.328085 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.329648 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.329654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.331506 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.331521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.332673 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.332686 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.333055 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.333917 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:27.333954 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:27.337113 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:27.340409 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:27.340418 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:27.340434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:27.341122 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:27.341618 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:27.341625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:27.341628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:27.341880 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:27.342451 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:27.342462 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:27.342816 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.343068 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.343621 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":420,"threads_started":1}
I20260620 18:29:27.354280 1718 heartbeater.cc:503] Master 127.0.97.62:39939 was elected leader, sending a full tablet report...
I20260620 18:29:27.369860 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.354949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.97.3:36115)
0620 18:29:27.356873 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.356898 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.357079 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.357464 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.357468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.357937 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:27.358212 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.358220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.359344 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.359350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.361088 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.361102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.361829 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.361835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.362025 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.362750 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:27.362766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:27.363498 (+ 732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:27.367123 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:27.367130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:27.367134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:27.367732 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:27.368105 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:27.368110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:27.368115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:27.368286 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:27.369062 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:27.369065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:27.369209 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.369315 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.369549 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"thread_start_us":1462,"threads_started":1}
I20260620 18:29:27.370301 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.354487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36115 (local address 127.0.97.62:39939)
0620 18:29:27.354760 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.354768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.356981 (+ 2213us) server_negotiation.cc:407] Connection header received
0620 18:29:27.357246 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.357251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.357330 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.357434 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:27.358346 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.359209 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.361245 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.361652 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.361792 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.363730 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:27.363736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:27.363741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:27.363777 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:27.367004 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.367858 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.367864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.367867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.367998 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.368443 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.368451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.368457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.368933 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:27.369123 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.369504 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.369804 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 18:29:27.388280 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35933
I20260620 18:29:27.388371 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35933 every 8 connection(s)
I20260620 18:29:27.389515 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:27.391666 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39939
I20260620 18:29:27.391950 1996 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:27.392639 1996 heartbeater.cc:511] Master 127.0.97.62:39939 requested a full tablet report, sending...
I20260620 18:29:27.392999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 1861
I20260620 18:29:27.393242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:27.393787 388 external_mini_cluster.cc:1428] Setting key 2b7ac974344a8c4d909d42d8ee935854
I20260620 18:29:27.394212 1511 ts_manager.cc:194] Registered new tserver with Master: 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3:35933)
I20260620 18:29:27.395443 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40849
I20260620 18:29:27.397121 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:29:27 dist-test-slave-mcnm krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:29:27 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:27.444095 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.429388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.97.62:39939)
0620 18:29:27.429598 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.429619 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.429752 (+ 133us) server_negotiation.cc:407] Connection header received
0620 18:29:27.429991 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.429995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.430058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.430167 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:27.431762 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.432560 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.433880 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.434323 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.434470 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.438561 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:27.438567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:27.438570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:27.438608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:27.441384 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.442059 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.442071 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.442074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.442201 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:27.442599 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:27.442605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:27.442609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:27.442873 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:27.443122 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.443426 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.443676 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:29:27.448259 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:27.455147 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:27.484185 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.468550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43903 (local address 127.0.0.1:37370)
0620 18:29:27.474927 (+ 6377us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.474955 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.475121 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.476221 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.476229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.476269 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:27.476743 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.476764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.478750 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.478756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.481383 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.481398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.483222 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.483231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.483444 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.483532 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.483667 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.483837 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6173,"thread_start_us":957,"threads_started":1}
I20260620 18:29:27.484295 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.469248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.97.2:43903)
0620 18:29:27.474437 (+ 5189us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.475325 (+ 888us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.475356 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:29:27.475578 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.475591 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.475932 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.476174 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:27.477007 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.478579 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.481590 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.483045 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.483190 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.483405 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.483748 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.483953 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4979,"thread_start_us":4812,"threads_started":1}
I20260620 18:29:27.486369 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.467824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46883 (local address 127.0.0.1:36580)
0620 18:29:27.469434 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.469492 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.469701 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.476607 (+ 6906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.476614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.476643 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:27.477157 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.477171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.480290 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.480296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.482509 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.482524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.485563 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.485574 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.485749 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.485781 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.485872 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.485991 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":1,"thread_start_us":6712,"threads_started":1}
I20260620 18:29:27.486434 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.468085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.0.97.1:46883)
0620 18:29:27.475901 (+ 7816us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.475907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.475929 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:29:27.476040 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.476049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.476312 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.477396 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:27.477687 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.480127 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.482711 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.485342 (+ 2631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.485485 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.485698 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.485941 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.486095 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7665,"thread_start_us":7548,"threads_started":1}
I20260620 18:29:27.488389 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 0d122cad48914259b763adcba4c4b078 (DEFAULT_TABLE table=test-table [id=6f79f537ab124869817308423e88245e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.489136 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.475218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.97.3:35933)
0620 18:29:27.481625 (+ 6407us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.481633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.481988 (+ 355us) server_negotiation.cc:407] Connection header received
0620 18:29:27.482192 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.482197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.482369 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.482548 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:27.483129 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.484721 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.486922 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.488100 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.488243 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.488457 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.488650 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.488811 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6226,"thread_start_us":6051,"threads_started":1}
I20260620 18:29:27.489136 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.476594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35933 (local address 127.0.0.1:37502)
0620 18:29:27.481861 (+ 5267us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.481879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.482021 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.482542 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.482546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.482568 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:27.482943 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.482952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.484873 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.484880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.486716 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.486730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.488287 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.488293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.488460 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.488495 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.488613 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.488727 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5149,"mutex_wait_us":1,"thread_start_us":5014,"threads_started":1}
I20260620 18:29:27.490221 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d122cad48914259b763adcba4c4b078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.490264 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 0d122cad48914259b763adcba4c4b078 (DEFAULT_TABLE table=test-table [id=6f79f537ab124869817308423e88245e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.491729 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d122cad48914259b763adcba4c4b078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.494271 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 0d122cad48914259b763adcba4c4b078 (DEFAULT_TABLE table=test-table [id=6f79f537ab124869817308423e88245e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.495285 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d122cad48914259b763adcba4c4b078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.506054 2015 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Bootstrap starting.
I20260620 18:29:27.509084 2015 tablet_bootstrap.cc:654] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.510058 2015 log.cc:824] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:27.511337 2016 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Bootstrap starting.
I20260620 18:29:27.513192 2015 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: No bootstrap required, opened a new log
I20260620 18:29:27.513442 2015 ts_tablet_manager.cc:1397] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 18:29:27.514740 2016 tablet_bootstrap.cc:654] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.515944 2016 log.cc:824] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:27.516981 2017 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Bootstrap starting.
I20260620 18:29:27.519232 2016 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: No bootstrap required, opened a new log
I20260620 18:29:27.519548 2016 ts_tablet_manager.cc:1397] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:29:27.520653 2017 tablet_bootstrap.cc:654] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.521782 2017 log.cc:824] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:27.523876 2015 raft_consensus.cc:348] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.524322 2015 raft_consensus.cc:374] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.524431 2015 raft_consensus.cc:729] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.524742 2017 tablet_bootstrap.cc:492] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: No bootstrap required, opened a new log
I20260620 18:29:27.525028 2017 ts_tablet_manager.cc:1397] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:29:27.525038 2015 consensus_queue.cc:260] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.526216 1996 heartbeater.cc:503] Master 127.0.97.62:39939 was elected leader, sending a full tablet report...
I20260620 18:29:27.526799 2015 ts_tablet_manager.cc:1428] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:29:27.530428 2016 raft_consensus.cc:348] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.530886 2016 raft_consensus.cc:374] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.531037 2016 raft_consensus.cc:729] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0647823b840f44d9aa08117b0397a1ae, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.531636 2016 consensus_queue.cc:260] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.532691 1857 heartbeater.cc:503] Master 127.0.97.62:39939 was elected leader, sending a full tablet report...
I20260620 18:29:27.533233 2016 ts_tablet_manager.cc:1428] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260620 18:29:27.535271 2017 raft_consensus.cc:348] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.535629 2017 raft_consensus.cc:374] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.535704 2017 raft_consensus.cc:729] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967dddd973c542039f40bc27b0277399, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.536525 2017 consensus_queue.cc:260] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.537897 2017 ts_tablet_manager.cc:1428] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260620 18:29:27.595623 2021 raft_consensus.cc:482] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:27.595956 2021 raft_consensus.cc:504] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
W20260620 18:29:27.596812 1719 tablet.cc:2369] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:27.598022 2021 leader_election.cc:288] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:27.603536 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.598467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51125 (local address 127.0.97.2:43903)
0620 18:29:27.598685 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.598692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.598799 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:29:27.599004 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.599009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.599079 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.599182 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:27.599595 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.600315 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.601869 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.602643 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.602792 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.602895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.603142 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.603265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:29:27.603509 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.598506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43903 (local address 127.0.97.3:51125)
0620 18:29:27.598685 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.598705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.598860 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.599210 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.599215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.599249 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:27.599438 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.599449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.600443 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.600451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.601711 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.601722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.602802 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.602807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.602989 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.603013 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.603120 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.603250 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:29:27.604394 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.598551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44747 (local address 127.0.97.1:46883)
0620 18:29:27.598727 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.598732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.599587 (+ 855us) server_negotiation.cc:407] Connection header received
0620 18:29:27.599783 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.599794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.599861 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.599968 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:27.600446 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.601139 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.602728 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.603470 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.603669 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.603772 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.604045 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.604164 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:29:27.604517 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.598505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46883 (local address 127.0.97.3:44747)
0620 18:29:27.599456 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:27.599471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:27.599653 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:27.599980 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:27.599985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:27.600005 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:27.600296 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.600303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.601276 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.601281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.602579 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:27.602589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.603740 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.603746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:27.603896 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.603918 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:29:27.604021 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:29:27.604146 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":211,"thread_start_us":380,"threads_started":1}
I20260620 18:29:27.604820 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d122cad48914259b763adcba4c4b078" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae" is_pre_election: true
I20260620 18:29:27.605288 1807 raft_consensus.cc:2454] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:27.605708 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d122cad48914259b763adcba4c4b078" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399" is_pre_election: true
I20260620 18:29:27.605966 1878 leader_election.cc:302] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0647823b840f44d9aa08117b0397a1ae, 185f50f120244da3b46afd0b6fab1f6d; no voters:
I20260620 18:29:27.606194 1673 raft_consensus.cc:2454] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:27.606353 2021 raft_consensus.cc:2781] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:27.606493 2021 raft_consensus.cc:482] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:27.606612 2021 raft_consensus.cc:3037] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:27.609292 2021 raft_consensus.cc:504] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.609845 2021 leader_election.cc:288] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Requested vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:27.610373 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d122cad48914259b763adcba4c4b078" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399"
I20260620 18:29:27.610383 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d122cad48914259b763adcba4c4b078" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae"
I20260620 18:29:27.610575 1673 raft_consensus.cc:3037] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:27.610635 1807 raft_consensus.cc:3037] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Advancing to term 1
W20260620 18:29:27.610872 1858 tablet.cc:2369] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:27.613059 1673 raft_consensus.cc:2454] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:27.613348 1807 raft_consensus.cc:2454] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:27.613516 1877 leader_election.cc:302] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185f50f120244da3b46afd0b6fab1f6d, 967dddd973c542039f40bc27b0277399; no voters:
I20260620 18:29:27.613813 2021 raft_consensus.cc:2781] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:27.614225 2021 raft_consensus.cc:686] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 1 LEADER]: Becoming Leader. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Running, Role: LEADER
I20260620 18:29:27.614761 2021 consensus_queue.cc:237] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:27.619081 1510 catalog_manager.cc:5495] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } health_report { overall_health: UNKNOWN } } }
W20260620 18:29:27.641076 1997 tablet.cc:2369] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:27.675333 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:27.670003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.0.97.3:35933)
0620 18:29:27.670181 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:27.670186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:27.670255 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:29:27.670527 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:27.670531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:27.670616 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:27.670733 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:27.671037 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.671811 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.673146 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:27.673571 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:27.673683 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:27.674288 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:27.674805 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:27.674918 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:27.675073 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:29:27.675136 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:29:27.682902 1807 raft_consensus.cc:1261] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Refusing update from remote peer 185f50f120244da3b46afd0b6fab1f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:27.682956 1673 raft_consensus.cc:1261] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Refusing update from remote peer 185f50f120244da3b46afd0b6fab1f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:27.683847 2025 consensus_queue.cc:1035] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:27.684157 2021 consensus_queue.cc:1035] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:27.717348 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:27.717604 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:27.721347 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Sending DeleteTablet for 3 replicas of tablet 0d122cad48914259b763adcba4c4b078
I20260620 18:29:27.722997 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d122cad48914259b763adcba4c4b078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37370
I20260620 18:29:27.723039 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d122cad48914259b763adcba4c4b078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36580
I20260620 18:29:27.723203 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d122cad48914259b763adcba4c4b078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37502
I20260620 18:29:27.725960 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:27.727600 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:27.737491 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d (DEFAULT_TABLE table=test-table [id=830232888f7f458297cdd1248dab631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.737972 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d (DEFAULT_TABLE table=test-table [id=830232888f7f458297cdd1248dab631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.738099 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f94aee0b6a48da9b63f98bb22a464d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.739012 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f94aee0b6a48da9b63f98bb22a464d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.739179 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d (DEFAULT_TABLE table=test-table [id=830232888f7f458297cdd1248dab631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:27.739907 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f94aee0b6a48da9b63f98bb22a464d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:27.745118 2016 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Bootstrap starting.
I20260620 18:29:27.745314 2015 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Bootstrap starting.
I20260620 18:29:27.746778 2016 tablet_bootstrap.cc:654] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.747633 2015 tablet_bootstrap.cc:654] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.750555 2040 tablet_replica.cc:330] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: stopping tablet replica
I20260620 18:29:27.751119 2040 raft_consensus.cc:2229] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:27.752156 2039 tablet_replica.cc:330] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: stopping tablet replica
I20260620 18:29:27.752682 2039 raft_consensus.cc:2229] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:27.753235 2039 raft_consensus.cc:2258] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:27.756585 2040 raft_consensus.cc:2258] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:27.759584 2015 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: No bootstrap required, opened a new log
I20260620 18:29:27.759802 2015 ts_tablet_manager.cc:1397] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260620 18:29:27.761304 2017 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Bootstrap starting.
I20260620 18:29:27.761912 2015 raft_consensus.cc:348] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.762204 2016 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: No bootstrap required, opened a new log
I20260620 18:29:27.762254 2015 raft_consensus.cc:374] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.762365 2016 ts_tablet_manager.cc:1397] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.009s
I20260620 18:29:27.762493 2015 raft_consensus.cc:729] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.763140 2016 raft_consensus.cc:348] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.763374 2016 raft_consensus.cc:374] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.763473 2017 tablet_bootstrap.cc:654] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:27.763471 2016 raft_consensus.cc:729] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0647823b840f44d9aa08117b0397a1ae, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.763788 2016 consensus_queue.cc:260] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.765255 2016 ts_tablet_manager.cc:1428] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:27.770205 2017 tablet_bootstrap.cc:492] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: No bootstrap required, opened a new log
I20260620 18:29:27.770361 2017 ts_tablet_manager.cc:1397] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260620 18:29:27.770783 2038 tablet_replica.cc:330] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: stopping tablet replica
I20260620 18:29:27.771104 2040 ts_tablet_manager.cc:1905] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:27.771159 2017 raft_consensus.cc:348] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.771395 2017 raft_consensus.cc:374] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:27.771497 2017 raft_consensus.cc:729] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967dddd973c542039f40bc27b0277399, State: Initialized, Role: FOLLOWER
I20260620 18:29:27.771341 2038 raft_consensus.cc:2229] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:27.771682 2015 consensus_queue.cc:260] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.771828 2017 consensus_queue.cc:260] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:27.772218 2038 raft_consensus.cc:2258] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:27.772354 2017 ts_tablet_manager.cc:1428] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:29:27.772401 2015 ts_tablet_manager.cc:1428] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20260620 18:29:27.774459 2038 ts_tablet_manager.cc:1905] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:27.781351 2038 ts_tablet_manager.cc:1918] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:27.781371 2039 ts_tablet_manager.cc:1905] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:27.781518 2038 log.cc:1196] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/0d122cad48914259b763adcba4c4b078
I20260620 18:29:27.782043 2038 ts_tablet_manager.cc:1939] T 0d122cad48914259b763adcba4c4b078 P 0647823b840f44d9aa08117b0397a1ae: Deleting consensus metadata
I20260620 18:29:27.783135 2040 ts_tablet_manager.cc:1918] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:27.783383 2040 log.cc:1196] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/0d122cad48914259b763adcba4c4b078
I20260620 18:29:27.783999 2040 ts_tablet_manager.cc:1939] T 0d122cad48914259b763adcba4c4b078 P 185f50f120244da3b46afd0b6fab1f6d: Deleting consensus metadata
I20260620 18:29:27.784255 1496 catalog_manager.cc:4841] TS 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903): tablet 0d122cad48914259b763adcba4c4b078 (table test-table [id=6f79f537ab124869817308423e88245e]) successfully deleted
I20260620 18:29:27.785552 1495 catalog_manager.cc:4841] TS 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3:35933): tablet 0d122cad48914259b763adcba4c4b078 (table test-table [id=6f79f537ab124869817308423e88245e]) successfully deleted
I20260620 18:29:27.788539 2039 ts_tablet_manager.cc:1918] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:27.788705 2039 log.cc:1196] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/0d122cad48914259b763adcba4c4b078
I20260620 18:29:27.789114 2039 ts_tablet_manager.cc:1939] T 0d122cad48914259b763adcba4c4b078 P 967dddd973c542039f40bc27b0277399: Deleting consensus metadata
I20260620 18:29:27.790397 1495 catalog_manager.cc:4841] TS 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883): tablet 0d122cad48914259b763adcba4c4b078 (table test-table [id=6f79f537ab124869817308423e88245e]) successfully deleted
I20260620 18:29:28.085041 2021 raft_consensus.cc:482] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:28.085299 2021 raft_consensus.cc:504] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:28.086138 2021 leader_election.cc:288] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:28.086764 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f94aee0b6a48da9b63f98bb22a464d" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399" is_pre_election: true
I20260620 18:29:28.086788 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f94aee0b6a48da9b63f98bb22a464d" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae" is_pre_election: true
I20260620 18:29:28.087041 1673 raft_consensus.cc:2454] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:28.087049 1807 raft_consensus.cc:2454] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:28.087584 1877 leader_election.cc:302] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185f50f120244da3b46afd0b6fab1f6d, 967dddd973c542039f40bc27b0277399; no voters:
I20260620 18:29:28.087891 2021 raft_consensus.cc:2781] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:28.088029 2021 raft_consensus.cc:482] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:28.088164 2021 raft_consensus.cc:3037] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.090569 2021 raft_consensus.cc:504] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:28.091220 2021 leader_election.cc:288] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Requested vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:28.091763 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f94aee0b6a48da9b63f98bb22a464d" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399"
I20260620 18:29:28.091777 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f94aee0b6a48da9b63f98bb22a464d" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae"
I20260620 18:29:28.091980 1807 raft_consensus.cc:3037] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.091971 1673 raft_consensus.cc:3037] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.094655 1673 raft_consensus.cc:2454] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:28.094975 1807 raft_consensus.cc:2454] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:28.095139 1877 leader_election.cc:302] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185f50f120244da3b46afd0b6fab1f6d, 967dddd973c542039f40bc27b0277399; no voters:
I20260620 18:29:28.095605 2021 raft_consensus.cc:2781] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:28.095785 2021 raft_consensus.cc:686] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 1 LEADER]: Becoming Leader. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Running, Role: LEADER
I20260620 18:29:28.096076 2021 consensus_queue.cc:237] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } }
I20260620 18:29:28.099189 1511 catalog_manager.cc:5495] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } health_report { overall_health: HEALTHY } } }
Jun 20 18:29:28 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:28.184306 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.170390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39939 (local address 127.0.0.1:38720)
0620 18:29:28.171146 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:28.171166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:28.171301 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:28.171747 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:28.171750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:28.172419 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:28.172626 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.172634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.173730 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.173734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.174807 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.174821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.175731 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.175740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.175985 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.176709 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:28.176726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:28.178968 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:28.181860 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:28.181872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:28.181880 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:28.182262 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:28.182610 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:28.182614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:28.182618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:28.182762 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:28.183240 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:28.183247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:28.183385 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:29:28.183577 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 18:29:28.183852 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":421,"threads_started":1}
I20260620 18:29:28.184358 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.170430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.97.62:39939)
0620 18:29:28.171170 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.171174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.171244 (+ 70us) server_negotiation.cc:407] Connection header received
0620 18:29:28.171537 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.171541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.171601 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.171697 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.172783 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.173586 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.174946 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.175460 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.175628 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.179149 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.179153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.179157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.179182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.181719 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.182374 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.182381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.182385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.182500 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.182862 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.182866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.182870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.183114 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.183249 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.183665 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.183974 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":624,"thread_start_us":477,"threads_started":1}
I20260620 18:29:28.189273 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:29:28.190315 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:28.197120 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 6598d57968de4e6f993cda54292a7a09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cb6b6e97f3340f99a81a706ee49c254]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:28.197162 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 6598d57968de4e6f993cda54292a7a09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cb6b6e97f3340f99a81a706ee49c254]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:28.197693 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6598d57968de4e6f993cda54292a7a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:28.197691 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6598d57968de4e6f993cda54292a7a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:28.197557 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 6598d57968de4e6f993cda54292a7a09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cb6b6e97f3340f99a81a706ee49c254]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:28.198101 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6598d57968de4e6f993cda54292a7a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:28.203071 2015 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: Bootstrap starting.
I20260620 18:29:28.204420 2015 tablet_bootstrap.cc:654] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:28.205116 2017 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: Bootstrap starting.
I20260620 18:29:28.205438 2016 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: Bootstrap starting.
I20260620 18:29:28.206998 2017 tablet_bootstrap.cc:654] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:28.207183 2016 tablet_bootstrap.cc:654] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:28.208068 2015 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: No bootstrap required, opened a new log
I20260620 18:29:28.208230 2015 ts_tablet_manager.cc:1397] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:28.209194 2015 raft_consensus.cc:348] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.209527 2015 raft_consensus.cc:374] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:28.209627 2015 raft_consensus.cc:729] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Initialized, Role: FOLLOWER
I20260620 18:29:28.209906 2015 consensus_queue.cc:260] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.210351 2021 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.210675 2015 ts_tablet_manager.cc:1428] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:29:28.210671 2021 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.211354 2017 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: No bootstrap required, opened a new log
I20260620 18:29:28.211529 2017 ts_tablet_manager.cc:1397] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 18:29:28.211807 2016 tablet_bootstrap.cc:492] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: No bootstrap required, opened a new log
I20260620 18:29:28.211961 2016 ts_tablet_manager.cc:1397] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 18:29:28.212247 2017 raft_consensus.cc:348] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.212484 2017 raft_consensus.cc:374] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:28.212555 2017 raft_consensus.cc:729] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967dddd973c542039f40bc27b0277399, State: Initialized, Role: FOLLOWER
I20260620 18:29:28.212699 2016 raft_consensus.cc:348] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.212785 2017 consensus_queue.cc:260] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.212986 2016 raft_consensus.cc:374] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:28.213063 2016 raft_consensus.cc:729] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0647823b840f44d9aa08117b0397a1ae, State: Initialized, Role: FOLLOWER
I20260620 18:29:28.213349 2017 ts_tablet_manager.cc:1428] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:29:28.213231 2023 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.213318 2016 consensus_queue.cc:260] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.213560 2023 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.213749 2022 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.214104 2022 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.214445 2016 ts_tablet_manager.cc:1428] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:29:28.588661 2021 consensus_queue.cc:1035] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:28.593750 2021 consensus_queue.cc:1035] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:28.617689 2022 raft_consensus.cc:482] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:28.617924 2022 raft_consensus.cc:504] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.619271 2021 raft_consensus.cc:482] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:28.619388 2022 leader_election.cc:288] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3:35933)
I20260620 18:29:28.619494 2021 raft_consensus.cc:504] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.620188 2021 leader_election.cc:288] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:28.620687 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399" is_pre_election: true
I20260620 18:29:28.620770 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae" is_pre_election: true
I20260620 18:29:28.620983 1807 raft_consensus.cc:2454] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:28.621425 1673 raft_consensus.cc:2454] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 0.
I20260620 18:29:28.621834 1878 leader_election.cc:302] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0647823b840f44d9aa08117b0397a1ae, 185f50f120244da3b46afd0b6fab1f6d; no voters:
I20260620 18:29:28.622121 2021 raft_consensus.cc:2781] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:28.622233 2021 raft_consensus.cc:482] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:28.622473 2021 raft_consensus.cc:3037] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.625265 2021 raft_consensus.cc:504] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.626361 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0647823b840f44d9aa08117b0397a1ae"
I20260620 18:29:28.626329 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "185f50f120244da3b46afd0b6fab1f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399"
I20260620 18:29:28.626603 1807 raft_consensus.cc:3037] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.626605 1673 raft_consensus.cc:3037] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:28.628521 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.619794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57439 (local address 127.0.97.1:46883)
0620 18:29:28.621503 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.621511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.621555 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:29:28.621631 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.621639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.621729 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.621838 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:28.622781 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.623921 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.626225 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.627380 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.627558 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.627661 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.628026 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.628176 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":2929,"threads_started":1}
I20260620 18:29:28.629276 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.619699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46883 (local address 127.0.97.2:57439)
0620 18:29:28.620339 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:28.620361 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:28.620493 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:28.622130 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:28.622138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:28.622172 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:28.622560 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.622575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.624063 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.624075 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.626097 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.626110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.627569 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.627574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.627798 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.627840 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:29:28.627948 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:29:28.628853 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":743,"thread_start_us":317,"threads_started":1}
I20260620 18:29:28.629698 2021 leader_election.cc:288] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Requested vote from peers 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883), 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903)
I20260620 18:29:28.629997 1807 raft_consensus.cc:2454] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:28.630048 1673 raft_consensus.cc:2454] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f50f120244da3b46afd0b6fab1f6d in term 1.
I20260620 18:29:28.630208 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "0647823b840f44d9aa08117b0397a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967dddd973c542039f40bc27b0277399" is_pre_election: true
I20260620 18:29:28.630808 1878 leader_election.cc:302] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0647823b840f44d9aa08117b0397a1ae, 185f50f120244da3b46afd0b6fab1f6d; no voters:
I20260620 18:29:28.631073 1672 raft_consensus.cc:2379] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0647823b840f44d9aa08117b0397a1ae in current term 1: Already voted for candidate 185f50f120244da3b46afd0b6fab1f6d in this term.
I20260620 18:29:28.631096 2021 raft_consensus.cc:2781] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:28.631475 2021 raft_consensus.cc:686] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [term 1 LEADER]: Becoming Leader. State: Replica: 185f50f120244da3b46afd0b6fab1f6d, State: Running, Role: LEADER
I20260620 18:29:28.631806 2021 consensus_queue.cc:237] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } }
I20260620 18:29:28.632730 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.620840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36027 (local address 127.0.97.3:35933)
0620 18:29:28.622599 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.622604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.623328 (+ 724us) server_negotiation.cc:407] Connection header received
0620 18:29:28.623467 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.623473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.623568 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.623665 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:28.624359 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.625457 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.629953 (+ 4496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.631663 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.631821 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.631917 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.632252 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.632382 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628,"thread_start_us":1409,"threads_started":1}
I20260620 18:29:28.632719 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.620482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35933 (local address 127.0.97.2:36027)
0620 18:29:28.623210 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:28.623228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:28.623348 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:28.623697 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:28.623702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:28.623773 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:28.624168 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.624178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.625573 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.625579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.629772 (+ 4193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.629787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.631857 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.631863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.632052 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.632108 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:29:28.632224 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:29:28.632352 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2611,"thread_start_us":2431,"threads_started":1}
I20260620 18:29:28.633688 2025 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: TxnStatusTablet state changed. Reason: New leader 185f50f120244da3b46afd0b6fab1f6d. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.633800 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6598d57968de4e6f993cda54292a7a09" candidate_uuid: "0647823b840f44d9aa08117b0397a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185f50f120244da3b46afd0b6fab1f6d" is_pre_election: true
I20260620 18:29:28.634073 2025 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:28.634639 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:29:28.634862 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:29:28.634829 1738 leader_election.cc:302] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0647823b840f44d9aa08117b0397a1ae; no voters: 185f50f120244da3b46afd0b6fab1f6d, 967dddd973c542039f40bc27b0277399
I20260620 18:29:28.635257 2022 raft_consensus.cc:2726] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:29:28.635418 1511 catalog_manager.cc:5495] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } health_report { overall_health: UNKNOWN } } }
I20260620 18:29:28.674260 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.668036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35933 (local address 127.0.0.1:37508)
0620 18:29:28.668269 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:28.668290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:28.668431 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:28.668858 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:28.668862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:28.668872 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:28.669013 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.669020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.670480 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.670485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.671665 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.671678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.673050 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.673055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.673266 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.673321 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:28.673765 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:28.673803 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:29:28.673882 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:29:28.674022 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:29:28.674392 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.668031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.97.3:35933)
0620 18:29:28.668245 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.668251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.668376 (+ 125us) server_negotiation.cc:407] Connection header received
0620 18:29:28.668596 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.668601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.668690 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.668792 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:28.669154 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.670316 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.671848 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.672888 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.673017 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.673508 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:28.673640 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:28.673739 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.673934 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.674015 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:29:28.678181 1807 raft_consensus.cc:1261] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Refusing update from remote peer 185f50f120244da3b46afd0b6fab1f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:28.678196 1672 raft_consensus.cc:1261] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Refusing update from remote peer 185f50f120244da3b46afd0b6fab1f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:28.678768 2025 consensus_queue.cc:1035] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:28.679055 2021 consensus_queue.cc:1035] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:28.681859 2023 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: TxnStatusTablet state changed. Reason: New leader 185f50f120244da3b46afd0b6fab1f6d. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.682159 2023 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.683504 2056 mvcc.cc:205] Tried to move back new op lower bound from 7298990770900168704 to 7298990770718736384. Current Snapshot: MvccSnapshot[applied={T|T < 7298990770900168704}]
I20260620 18:29:28.685571 2022 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: TxnStatusTablet state changed. Reason: New leader 185f50f120244da3b46afd0b6fab1f6d. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.685860 2022 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.687070 2023 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.687314 2023 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 967dddd973c542039f40bc27b0277399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.689661 2057 mvcc.cc:205] Tried to move back new op lower bound from 7298990770900168704 to 7298990770718736384. Current Snapshot: MvccSnapshot[applied={T|T < 7298990770900168704}]
I20260620 18:29:28.696136 2022 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.696436 2022 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 0647823b840f44d9aa08117b0397a1ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:28.696427 2025 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.696671 2025 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:28.697091 2021 tablet_replica.cc:439] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "185f50f120244da3b46afd0b6fab1f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967dddd973c542039f40bc27b0277399" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46883 } } peers { permanent_uuid: "185f50f120244da3b46afd0b6fab1f6d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35933 } } peers { permanent_uuid: "0647823b840f44d9aa08117b0397a1ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43903 } } }
I20260620 18:29:28.697352 2021 tablet_replica.cc:442] T 6598d57968de4e6f993cda54292a7a09 P 185f50f120244da3b46afd0b6fab1f6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:28.719357 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.712446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58273 (local address 127.0.97.3:35933)
0620 18:29:28.712623 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.712629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.713139 (+ 510us) server_negotiation.cc:407] Connection header received
0620 18:29:28.713333 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.713338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.713412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.713500 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:28.714006 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.715193 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.717273 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.717963 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.718077 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.718564 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:28.718708 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:28.718803 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.718942 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.719016 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:29:28.719480 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.712351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35933 (local address 127.0.97.3:58273)
0620 18:29:28.713025 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:28.713044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:28.713187 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:28.713533 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:28.713538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:28.713595 (+ 57us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:28.713861 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.713873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.715337 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.715343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.717099 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:28.717113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.718110 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.718117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:28.718308 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.718382 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:28.718817 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:28.718827 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:29:28.718914 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:29:28.719050 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":397,"threads_started":1}
I20260620 18:29:28.784850 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:28.785066 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:28.788189 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ae5b6985ccc4c4ab06b8c19f524ae9a: Sending DeleteTablet for 3 replicas of tablet 65f94aee0b6a48da9b63f98bb22a464d
I20260620 18:29:28.788980 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37370
I20260620 18:29:28.789065 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36580
I20260620 18:29:28.789913 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f94aee0b6a48da9b63f98bb22a464d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37502
I20260620 18:29:28.800379 2078 tablet_replica.cc:330] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: stopping tablet replica
I20260620 18:29:28.800647 2078 raft_consensus.cc:2229] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:28.801029 2078 raft_consensus.cc:2258] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:28.801630 2079 tablet_replica.cc:330] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: stopping tablet replica
I20260620 18:29:28.801868 2079 raft_consensus.cc:2229] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:28.802194 2079 raft_consensus.cc:2258] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:28.803659 2079 ts_tablet_manager.cc:1905] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:28.805757 2080 tablet_replica.cc:330] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: stopping tablet replica
I20260620 18:29:28.806057 2080 raft_consensus.cc:2229] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:28.806578 2080 raft_consensus.cc:2258] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:28.807741 2080 ts_tablet_manager.cc:1905] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:28.811091 2079 ts_tablet_manager.cc:1918] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:28.811226 2079 log.cc:1196] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/65f94aee0b6a48da9b63f98bb22a464d
I20260620 18:29:28.811743 2079 ts_tablet_manager.cc:1939] T 65f94aee0b6a48da9b63f98bb22a464d P 967dddd973c542039f40bc27b0277399: Deleting consensus metadata
I20260620 18:29:28.812940 1495 catalog_manager.cc:4841] TS 967dddd973c542039f40bc27b0277399 (127.0.97.1:46883): tablet 65f94aee0b6a48da9b63f98bb22a464d (table test-table [id=830232888f7f458297cdd1248dab631b]) successfully deleted
I20260620 18:29:28.814105 2078 ts_tablet_manager.cc:1905] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:28.815980 2080 ts_tablet_manager.cc:1918] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:28.816185 2080 log.cc:1196] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/65f94aee0b6a48da9b63f98bb22a464d
I20260620 18:29:28.816671 2080 ts_tablet_manager.cc:1939] T 65f94aee0b6a48da9b63f98bb22a464d P 185f50f120244da3b46afd0b6fab1f6d: Deleting consensus metadata
I20260620 18:29:28.817857 1495 catalog_manager.cc:4841] TS 185f50f120244da3b46afd0b6fab1f6d (127.0.97.3:35933): tablet 65f94aee0b6a48da9b63f98bb22a464d (table test-table [id=830232888f7f458297cdd1248dab631b]) successfully deleted
I20260620 18:29:28.821239 2078 ts_tablet_manager.cc:1918] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:28.821391 2078 log.cc:1196] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/65f94aee0b6a48da9b63f98bb22a464d
I20260620 18:29:28.821884 2078 ts_tablet_manager.cc:1939] T 65f94aee0b6a48da9b63f98bb22a464d P 0647823b840f44d9aa08117b0397a1ae: Deleting consensus metadata
I20260620 18:29:28.823000 1496 catalog_manager.cc:4841] TS 0647823b840f44d9aa08117b0397a1ae (127.0.97.2:43903): tablet 65f94aee0b6a48da9b63f98bb22a464d (table test-table [id=830232888f7f458297cdd1248dab631b]) successfully deleted
Jun 20 18:29:28 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:28.834638 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.813276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.0.97.1:46883)
0620 18:29:28.813458 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.813465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.818160 (+ 4695us) server_negotiation.cc:407] Connection header received
0620 18:29:28.818358 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.818363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.818446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.818555 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.820403 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.821520 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.823278 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.823848 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.823987 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.828524 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.828537 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.828552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.828589 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.831743 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.832578 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.832585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.832587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.832712 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.833146 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.833150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.833152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.833567 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.833791 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.834137 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.834303 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:29:28.835418 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36584
I20260620 18:29:28.853562 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.838905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.0.97.62:39939)
0620 18:29:28.839949 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.839955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.839973 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:29:28.840031 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.840036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.840109 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.840231 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.841525 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.842730 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.844109 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.844800 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.844938 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.847536 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.847545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.847551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.847596 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.850730 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.851642 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.851648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.851651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.851795 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.852181 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.852187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.852191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.852459 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.852705 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.853009 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.853220 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":756,"threads_started":1}
W20260620 18:29:28.854218 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 18:29:28.860085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1582
I20260620 18:29:28.877506 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.861604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.0.97.1:46883)
0620 18:29:28.861772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.861777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.861891 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:29:28.862151 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.862157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.862255 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.862352 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.863859 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.864925 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.866789 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.867551 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.867669 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.870993 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.870999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.871005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.871042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.874350 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.875118 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.875123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.875127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.875262 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.875759 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.875764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.875768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.876091 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.876325 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.876694 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.876985 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:29:28.878003 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36594
W20260620 18:29:28.878597 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:28.878764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1582
I20260620 18:29:28.899016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1722
Jun 20 18:29:28 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:28.916175 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.899966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.0.97.2:43903)
0620 18:29:28.900562 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.900567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.900583 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:29:28.900638 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.900642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.900743 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.900891 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.902153 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.902961 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.904341 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.904826 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.904979 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.910112 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.910130 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.910147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.910194 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.913333 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.914114 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.914130 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.914134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.914268 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.914661 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.914667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.914670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.915021 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.915330 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.915610 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.915836 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
W20260620 18:29:28.917117 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37374
W20260620 18:29:28.917807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:28.918042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1722
I20260620 18:29:28.936546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1861
Jun 20 18:29:28 dist-test-slave-mcnm krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:28.955639 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:28.937353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.0.97.3:35933)
0620 18:29:28.937531 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:28.937536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:28.937553 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:29:28.937754 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:28.937758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:28.937825 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:28.937910 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:28.939631 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.940423 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.942204 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:28.942663 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:28.942764 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:28.948296 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:28.948310 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:28.948328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:28.948368 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:28.952126 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.952809 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.952824 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.952833 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.953007 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:28.953509 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:28.953516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:28.953521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:28.954147 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:28.954510 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:28.955023 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 18:29:28.955205 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:29:28.956391 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37522
W20260620 18:29:28.957091 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:28.957261 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1861
I20260620 18:29:28.980252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 1479
I20260620 18:29:28.981335 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:29.095000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 1479
2026-06-20T18:29:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5070 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:29 dist-test-slave-mcnm krb5kdc[2091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:29 dist-test-slave-mcnm krb5kdc[2091](info): set up 2 sockets
Jun 20 18:29:29 dist-test-slave-mcnm krb5kdc[2091](info): commencing operation
krb5kdc: starting...
W20260620 18:29:30.191982 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:30 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:30.377394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46791
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:30.694782 2104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:30.695068 2104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:30.695225 2104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:30.702376 2104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:30.702479 2104 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:29:30.702555 2104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:30.702651 2104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:30.702735 2104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:30.702852 2104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:30.712653 2104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46791
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:30.713954 2104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:30.715296 2104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:30.727113 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:30.727150 2112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:30.727794 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:30.728271 2104 server_base.cc:1034] running on GCE node
I20260620 18:29:30.729044 2104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:30.730399 2104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:30.731694 2104 hybrid_clock.cc:648] HybridClock initialized: now 1781980170731620 us; error 34 us; skew 500 ppm
I20260620 18:29:30.732051 2104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:30 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:30.735435 2104 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:30.737071 2104 webserver.cc:460] Webserver started at http://127.0.97.62:46277/ using document root <none> and password file <none>
I20260620 18:29:30.737769 2104 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:30.737856 2104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:30.738106 2104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:30.741079 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "335906ac91c44e4dbd994f052fdfb416"
format_stamp: "Formatted at 2026-06-20 18:29:30 on dist-test-slave-mcnm"
server_key: "9b8a290e86ee2be91030f00522535946"
server_key_iv: "bfe72a9cb0be1d4375b8d0a3ececbb30"
server_key_version: "encryptionkey@0"
I20260620 18:29:30.741803 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "335906ac91c44e4dbd994f052fdfb416"
format_stamp: "Formatted at 2026-06-20 18:29:30 on dist-test-slave-mcnm"
server_key: "9b8a290e86ee2be91030f00522535946"
server_key_iv: "bfe72a9cb0be1d4375b8d0a3ececbb30"
server_key_version: "encryptionkey@0"
I20260620 18:29:30.746903 2104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:29:30.750612 2119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:30.752071 2104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:30.752261 2104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "335906ac91c44e4dbd994f052fdfb416"
format_stamp: "Formatted at 2026-06-20 18:29:30 on dist-test-slave-mcnm"
server_key: "9b8a290e86ee2be91030f00522535946"
server_key_iv: "bfe72a9cb0be1d4375b8d0a3ececbb30"
server_key_version: "encryptionkey@0"
I20260620 18:29:30.752478 2104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:30.789930 2104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:30.794564 2104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:30.794838 2104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:30.817344 2104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46791
I20260620 18:29:30.817381 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46791 every 8 connection(s)
I20260620 18:29:30.818693 2104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:30.822386 2181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:30.825040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2104
I20260620 18:29:30.825235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:30.825999 388 external_mini_cluster.cc:1428] Setting key b1a00324acc401c33a1ada2f0879736c
I20260620 18:29:30.834947 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Bootstrap starting.
I20260620 18:29:30.838847 2181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:30.840241 2181 log.cc:824] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:29:30 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:30.844050 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: No bootstrap required, opened a new log
I20260620 18:29:30.851104 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:30.828118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.97.62:46791)
0620 18:29:30.829030 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:30.829041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:30.829081 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:29:30.830762 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:30.830784 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:30.831195 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:30.831622 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:30.833312 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:30.834768 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:30.836541 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:30.837346 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:30.837483 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:30.842943 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:30.842952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:30.842968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:30.843003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:30.846328 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:30.847356 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:30.847369 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:30.847376 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:30.847673 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:30.848112 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:30.848116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:30.848118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:30.848666 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:30.848877 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:30.849490 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 18:29:30.849993 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":496,"threads_started":1}
I20260620 18:29:30.853772 2181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } }
I20260620 18:29:30.854171 2181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:30.854288 2181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335906ac91c44e4dbd994f052fdfb416, State: Initialized, Role: FOLLOWER
I20260620 18:29:30.854874 2181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } }
I20260620 18:29:30.855142 2181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:30.855312 2181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:30.855504 2181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:30.857977 2181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } }
I20260620 18:29:30.858525 2181 leader_election.cc:302] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 335906ac91c44e4dbd994f052fdfb416; no voters:
I20260620 18:29:30.859133 2181 leader_election.cc:288] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:30.859359 2186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:30.860136 2186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [term 1 LEADER]: Becoming Leader. State: Replica: 335906ac91c44e4dbd994f052fdfb416, State: Running, Role: LEADER
I20260620 18:29:30.860678 2186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } }
I20260620 18:29:30.861455 2181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:30.863381 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335906ac91c44e4dbd994f052fdfb416. Latest consensus state: current_term: 1 leader_uuid: "335906ac91c44e4dbd994f052fdfb416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } } }
I20260620 18:29:30.863744 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:30.864640 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:30.864607 2187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "335906ac91c44e4dbd994f052fdfb416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335906ac91c44e4dbd994f052fdfb416" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46791 } } }
I20260620 18:29:30.864921 2187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:30.871093 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:30.881040 2192 catalog_manager.cc:1347] Generated new cluster ID: 94316b15c4714648ac8d63de9e861df6
I20260620 18:29:30.881161 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:30.896976 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:30.898792 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:30.914227 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Generated new TSK 0
I20260620 18:29:30.915390 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:30.979310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46791
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:31.305930 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:31.306262 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:31.306438 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:31.313426 2207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:31.313570 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:31.313738 2207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:29:31.313833 2207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:31.313946 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:31.323702 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:31.324923 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:31.326244 2207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:31.339385 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:31.339347 2213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:31.339664 2212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:31.340693 2207 server_base.cc:1034] running on GCE node
I20260620 18:29:31.341302 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:31.342284 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:31.343466 2207 hybrid_clock.cc:648] HybridClock initialized: now 1781980171343425 us; error 34 us; skew 500 ppm
I20260620 18:29:31.343847 2207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:31 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:31.347128 2207 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:31.348773 2207 webserver.cc:460] Webserver started at http://127.0.97.1:32957/ using document root <none> and password file <none>
I20260620 18:29:31.349479 2207 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:31.349610 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:31.349972 2207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:31.352989 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "3f461ea159ba4d2eac934c3b417f68a5"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "f8035b195de84cc91a80a2214313286a"
server_key_iv: "ffab175ec2e714439fbcdec8e08eb466"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.353693 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3f461ea159ba4d2eac934c3b417f68a5"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "f8035b195de84cc91a80a2214313286a"
server_key_iv: "ffab175ec2e714439fbcdec8e08eb466"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.359156 2207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:29:31.363093 2222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:31.364614 2207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:29:31.364763 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "3f461ea159ba4d2eac934c3b417f68a5"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "f8035b195de84cc91a80a2214313286a"
server_key_iv: "ffab175ec2e714439fbcdec8e08eb466"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.364955 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:31.401686 2207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:31.406179 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:31.406445 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:31.409153 2207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:31.409276 2207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:31.409395 2207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:31.409478 2207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:29:31 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:31.438553 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.410497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.1:52369)
0620 18:29:31.411393 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:31.411444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:31.413909 (+ 2465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:31.415277 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:31.415292 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:31.416394 (+ 1102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:31.417771 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.417792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.419755 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.419760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.422688 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.422707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.423760 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.423772 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.424166 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.425048 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:31.425096 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:31.428648 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:31.431939 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.431949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.431966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.432766 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.433205 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.433210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.433213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.433509 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.434562 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:31.434574 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:31.435319 (+ 745us) client_negotiation.cc:769] Sending connection context
0620 18:29:31.435688 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 18:29:31.436328 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":431,"threads_started":1}
I20260620 18:29:31.440133 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.409807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52369 (local address 127.0.97.62:46791)
0620 18:29:31.410484 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:31.410489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:31.411623 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 18:29:31.414323 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:31.414332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:31.414481 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:31.414599 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:31.418132 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.419479 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.422874 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.423427 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.423629 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.428886 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:31.428891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:31.428894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:31.428926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:31.431753 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.432900 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.432906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.432910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.433061 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.433895 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.433900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.433903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.434309 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:31.434456 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:31.439176 (+ 4720us) server_negotiation.cc:299] Negotiation successful
0620 18:29:31.439706 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":438,"threads_started":1}
I20260620 18:29:31.464605 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.447772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.1:52563)
0620 18:29:31.448612 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:31.448636 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:31.448839 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:31.449410 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:31.449416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:31.450033 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:31.450496 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.450512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.452125 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.452130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.454199 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.454217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.455661 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.455675 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.455895 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.456787 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:31.456804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:31.457945 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:31.461464 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.461472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.461478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.462185 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.462649 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.462655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.462659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.462811 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.463543 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:31.463549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:31.463809 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 18:29:31.463998 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 18:29:31.464266 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":490,"threads_started":1}
I20260620 18:29:31.465407 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.447675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52563 (local address 127.0.97.62:46791)
0620 18:29:31.447850 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:31.447856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:31.448720 (+ 864us) server_negotiation.cc:407] Connection header received
0620 18:29:31.449003 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:31.449010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:31.449105 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:31.449180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:31.450699 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.451916 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.454396 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.455099 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.455217 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.458182 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:31.458188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:31.458194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:31.458229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:31.461296 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.462318 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.462324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.462328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.462534 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.462939 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.462946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.462951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.463370 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:31.463571 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:31.464937 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0620 18:29:31.465151 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:29:31.480525 2207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37929
I20260620 18:29:31.480603 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37929 every 8 connection(s)
I20260620 18:29:31.481951 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:31.484256 2343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46791
I20260620 18:29:31.484517 2343 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:31.485168 2343 heartbeater.cc:511] Master 127.0.97.62:46791 requested a full tablet report, sending...
I20260620 18:29:31.487644 2136 ts_manager.cc:194] Registered new tserver with Master: 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929)
I20260620 18:29:31.490007 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52369
I20260620 18:29:31.491834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2207
I20260620 18:29:31.492038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:31.492655 388 external_mini_cluster.cc:1428] Setting key d229713377c266e330aa880b69390240
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:31.531126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46791
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:31.844190 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:31.844466 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:31.844621 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:31.851926 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:31.852257 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:31.852463 2347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:29:31.852551 2347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:31.852679 2347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:31.863025 2347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:31.864367 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:31.865705 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:31.878321 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:31.878518 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:31.878362 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:31.879465 2347 server_base.cc:1034] running on GCE node
I20260620 18:29:31.880151 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:31.881142 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:31.882386 2347 hybrid_clock.cc:648] HybridClock initialized: now 1781980171882309 us; error 76 us; skew 500 ppm
I20260620 18:29:31.882719 2347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:31 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:31.889233 2347 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:31.890712 2347 webserver.cc:460] Webserver started at http://127.0.97.2:39005/ using document root <none> and password file <none>
I20260620 18:29:31.891368 2347 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:31.891466 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:31.891803 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:31.894623 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "f74c3f0649eb4012930065159a800b7a"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "d3f3c57ce9b2af90368cc6c6838d1ed4"
server_key_iv: "fa893e68c74eb28abcceb4832362f49a"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.895349 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f74c3f0649eb4012930065159a800b7a"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "d3f3c57ce9b2af90368cc6c6838d1ed4"
server_key_iv: "fa893e68c74eb28abcceb4832362f49a"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.900453 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:29:31.903909 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:31.905376 2347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:29:31.905572 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "f74c3f0649eb4012930065159a800b7a"
format_stamp: "Formatted at 2026-06-20 18:29:31 on dist-test-slave-mcnm"
server_key: "d3f3c57ce9b2af90368cc6c6838d1ed4"
server_key_iv: "fa893e68c74eb28abcceb4832362f49a"
server_key_version: "encryptionkey@0"
I20260620 18:29:31.905800 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:31.936436 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:31.941238 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:31.941550 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:31.944537 2347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:31.944633 2347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:31.944792 2347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:31.944880 2347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:29:31 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:31.971067 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.944921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43735 (local address 127.0.97.62:46791)
0620 18:29:31.945080 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:31.945085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:31.946693 (+ 1608us) server_negotiation.cc:407] Connection header received
0620 18:29:31.948946 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:31.948956 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:31.949069 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:31.949190 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:31.951486 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.952528 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.956503 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.957130 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.957306 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.961844 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:31.961849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:31.961853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:31.961888 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:31.965866 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.966874 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.966881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.966884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.967169 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.968045 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.968053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.968060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.968496 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:31.968689 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:31.970169 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0620 18:29:31.970424 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:29:31.971876 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.945617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.2:43735)
0620 18:29:31.946509 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:31.946554 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:31.948623 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:31.949482 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:31.949493 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:31.950260 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:31.951255 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.951278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.952742 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.952752 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.956286 (+ 3534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.956310 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.957503 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.957513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.957738 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.958622 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:31.958647 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:31.961558 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:31.966073 (+ 4515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.966085 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.966106 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.966701 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.967351 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.967362 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.967367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.967833 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.968759 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:31.968773 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:31.969476 (+ 703us) client_negotiation.cc:769] Sending connection context
0620 18:29:31.970040 (+ 564us) client_negotiation.cc:240] Negotiation successful
0620 18:29:31.970524 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":372,"threads_started":1}
I20260620 18:29:31.996830 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.977752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55361 (local address 127.0.97.62:46791)
0620 18:29:31.977950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:31.977959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:31.979016 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 18:29:31.979147 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:31.979155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:31.979253 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:31.979359 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:31.981584 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.982281 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.984902 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.985662 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.985811 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.988730 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:31.988738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:31.988742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:31.988786 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:31.992571 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.993785 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.993791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.993797 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.993968 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:31.994484 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:31.994491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:31.994497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:31.994900 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:31.995094 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:31.995781 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 18:29:31.996011 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:29:31.997119 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:31.977895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.2:55361)
0620 18:29:31.978901 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:31.978921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:31.979045 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:31.979588 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:31.979595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:31.980769 (+ 1174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:31.981318 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.981338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.982430 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.982439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.984728 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:31.984753 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:31.985906 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:31.985916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:31.986215 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:31.987006 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:31.987037 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:31.988499 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:31.992746 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.992755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.992762 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.993639 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.994112 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:31.994120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:31.994127 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:31.994330 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:31.995175 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:31.995186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:31.995451 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 18:29:31.995666 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 18:29:31.996497 (+ 831us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":704,"threads_started":1}
I20260620 18:29:32.001543 2347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33801
I20260620 18:29:32.001562 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33801 every 8 connection(s)
I20260620 18:29:32.003733 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:32.006850 2482 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46791
I20260620 18:29:32.007105 2482 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:32.007884 2482 heartbeater.cc:511] Master 127.0.97.62:46791 requested a full tablet report, sending...
I20260620 18:29:32.009702 2136 ts_manager.cc:194] Registered new tserver with Master: f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801)
I20260620 18:29:32.009763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2347
I20260620 18:29:32.009914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:32.010478 388 external_mini_cluster.cc:1428] Setting key f9d9ef56c39885ba1ca6ececa9a734fe
I20260620 18:29:32.011056 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43735
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:32.051812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46791
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:32.368875 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:32.369244 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:32.369419 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:32.376832 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:32.376973 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:32.377128 2486 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:29:32.377218 2486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:29:32.377349 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:32.386864 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:32.388227 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:32.389554 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:32.402717 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:32.402709 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:32.402709 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:32.403806 2486 server_base.cc:1034] running on GCE node
I20260620 18:29:32.404407 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:32.405303 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:32.406492 2486 hybrid_clock.cc:648] HybridClock initialized: now 1781980172406467 us; error 61 us; skew 500 ppm
I20260620 18:29:32.406802 2486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:32 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:32.409916 2486 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:32.411551 2486 webserver.cc:460] Webserver started at http://127.0.97.3:33445/ using document root <none> and password file <none>
I20260620 18:29:32.412256 2486 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:32.412401 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:32.412712 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:32.415684 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "999831788cd8412696c34774394578c6"
format_stamp: "Formatted at 2026-06-20 18:29:32 on dist-test-slave-mcnm"
server_key: "c5ace2711efc862b320a6a25559097c9"
server_key_iv: "492a3775c4afc28dc93d24b4fa6c817e"
server_key_version: "encryptionkey@0"
I20260620 18:29:32.416484 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "999831788cd8412696c34774394578c6"
format_stamp: "Formatted at 2026-06-20 18:29:32 on dist-test-slave-mcnm"
server_key: "c5ace2711efc862b320a6a25559097c9"
server_key_iv: "492a3775c4afc28dc93d24b4fa6c817e"
server_key_version: "encryptionkey@0"
I20260620 18:29:32.421662 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:29:32.425076 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:32.426455 2486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:29:32.426625 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "999831788cd8412696c34774394578c6"
format_stamp: "Formatted at 2026-06-20 18:29:32 on dist-test-slave-mcnm"
server_key: "c5ace2711efc862b320a6a25559097c9"
server_key_iv: "492a3775c4afc28dc93d24b4fa6c817e"
server_key_version: "encryptionkey@0"
I20260620 18:29:32.426823 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:32.455502 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:32.460476 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:32.460788 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:32.463718 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:32.463848 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:32.463968 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:32.464048 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:29:32 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:32.493340 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.464571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34655 (local address 127.0.97.62:46791)
0620 18:29:32.464745 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.464750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.466144 (+ 1394us) server_negotiation.cc:407] Connection header received
0620 18:29:32.468607 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.468615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.468723 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.468896 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:32.470699 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.472023 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.474874 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.475494 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.475652 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.480473 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:32.480480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:32.480484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:32.480529 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:32.489123 (+ 8594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.490412 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.490420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.490425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.490616 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.491258 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.491263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.491266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.491661 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:32.491809 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.492740 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.492961 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:29:32.493417 2343 heartbeater.cc:503] Master 127.0.97.62:46791 was elected leader, sending a full tablet report...
I20260620 18:29:32.495829 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.465161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.3:34655)
0620 18:29:32.465979 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.466019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.468243 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.469074 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.469082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.469910 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:32.470527 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.470545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.472219 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.472230 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.474671 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.474691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.475751 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.475758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.476062 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.476919 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:32.476960 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:32.480192 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:32.489309 (+ 9117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.489322 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.489345 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.490242 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.490746 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.490751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.490756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.491099 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.491806 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:32.491815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:32.492267 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.492661 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.493225 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":368,"threads_started":1}
I20260620 18:29:32.508531 2486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42137
I20260620 18:29:32.508575 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42137 every 8 connection(s)
I20260620 18:29:32.510053 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:32.512425 2621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46791
I20260620 18:29:32.512673 2621 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:32.513290 2621 heartbeater.cc:511] Master 127.0.97.62:46791 requested a full tablet report, sending...
I20260620 18:29:32.514688 2136 ts_manager.cc:194] Registered new tserver with Master: 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:32.515822 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34655
I20260620 18:29:32.517491 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.503180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.97.3:38133)
0620 18:29:32.504057 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.504076 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.504227 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.504583 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.504588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.505200 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:32.505612 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.505625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.506703 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.506709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.508669 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.508687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.509470 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.509482 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.509681 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.510398 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:32.510415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:32.511673 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:32.514977 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.514982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.514986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.515460 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.515840 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.515847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.515851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.515991 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.516688 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:32.516691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:32.516848 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.516953 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.517183 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":1,"thread_start_us":453,"threads_started":1}
I20260620 18:29:32.517743 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.503085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38133 (local address 127.0.97.62:46791)
0620 18:29:32.503285 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.503292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.504158 (+ 866us) server_negotiation.cc:407] Connection header received
0620 18:29:32.504377 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.504382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.504452 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.504560 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:32.505753 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.506549 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.508865 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.509279 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.509421 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.511869 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:32.511893 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:32.511898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:32.511933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:32.514862 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.515604 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.515608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.515610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.515732 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.516135 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.516140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.516143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.516483 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:32.516616 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.517150 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.517381 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:29:32.519145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2486
I20260620 18:29:32.519310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:32.519896 388 external_mini_cluster.cc:1428] Setting key ef86c85b34d6ac011820400f7fbabde3
I20260620 18:29:32.522544 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:29:32 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:29:32 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:32.569402 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.554711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.0.97.62:46791)
0620 18:29:32.554941 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.554950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.555256 (+ 306us) server_negotiation.cc:407] Connection header received
0620 18:29:32.555494 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.555498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.555646 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.555753 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:32.557132 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.557863 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.559062 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.559488 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.559670 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.563618 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:32.563624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:32.563629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:32.563668 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:32.566300 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.567051 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.567059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.567063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.567212 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.567702 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.567710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.567718 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.568065 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:32.568329 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.568717 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.568967 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:29:32.573995 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:32.581884 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:32.609002 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.595030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.3:42137)
0620 18:29:32.596153 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.596165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.597050 (+ 885us) server_negotiation.cc:407] Connection header received
0620 18:29:32.597158 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.597168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.597510 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.597729 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:32.598947 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.601152 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.605999 (+ 4847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.607372 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.607548 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.607775 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.608515 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.608673 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":695,"threads_started":1}
I20260620 18:29:32.609107 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.594481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42137 (local address 127.0.0.1:57664)
0620 18:29:32.596625 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.596643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.596750 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.598117 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.598128 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.598168 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:32.598685 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.598701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.601360 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.601368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.604462 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.604492 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.607584 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.607590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.607777 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.607841 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.607978 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.608206 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2012,"thread_start_us":1874,"threads_started":1}
I20260620 18:29:32.613323 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.594151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37929 (local address 127.0.0.1:35584)
0620 18:29:32.594929 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.594975 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.595147 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.605773 (+ 10626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.605781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.605812 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:32.606257 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.606268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.607983 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.607988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.610742 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.610755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.612609 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.612614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.612724 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.612746 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.612852 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.612972 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":360,"threads_started":1}
I20260620 18:29:32.613319 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.594370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.0.97.1:37929)
0620 18:29:32.605115 (+ 10745us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.605127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.605162 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:29:32.605282 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.605292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.605567 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.605760 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:32.606473 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.607850 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.610925 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.612143 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.612265 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.612472 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.612900 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.613047 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10547,"thread_start_us":10387,"threads_started":1}
I20260620 18:29:32.612970 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf (DEFAULT_TABLE table=test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:32.614678 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.596752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.0.1:56510)
0620 18:29:32.605747 (+ 8995us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.605763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.605883 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.606579 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.606584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.606607 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:32.606927 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.606937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.608451 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.608458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.611369 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.611384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.613266 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.613272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.614156 (+ 884us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.614177 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.614271 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.614386 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8869,"thread_start_us":8755,"threads_started":1}
I20260620 18:29:32.614908 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.597640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.0.97.2:33801)
0620 18:29:32.605397 (+ 7757us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.605404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.606017 (+ 613us) server_negotiation.cc:407] Connection header received
0620 18:29:32.606147 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.606156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.606391 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.606575 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:32.607091 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.608324 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.611555 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.612801 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.612935 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.613126 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.614325 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.614506 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7619,"thread_start_us":7493,"threads_started":1}
I20260620 18:29:32.615116 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037f899e8f914b7bbaa85c1b993b82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.617000 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf (DEFAULT_TABLE table=test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:32.617400 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf (DEFAULT_TABLE table=test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:32.618309 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037f899e8f914b7bbaa85c1b993b82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.618515 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037f899e8f914b7bbaa85c1b993b82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.629550 2640 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Bootstrap starting.
I20260620 18:29:32.632783 2640 tablet_bootstrap.cc:654] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.633958 2642 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Bootstrap starting.
I20260620 18:29:32.633975 2640 log.cc:824] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:32.636368 2641 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Bootstrap starting.
I20260620 18:29:32.638455 2642 tablet_bootstrap.cc:654] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.639166 2641 tablet_bootstrap.cc:654] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.639984 2642 log.cc:824] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:32.641448 2641 log.cc:824] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:32.643205 2642 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: No bootstrap required, opened a new log
I20260620 18:29:32.643205 2640 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: No bootstrap required, opened a new log
I20260620 18:29:32.643462 2642 ts_tablet_manager.cc:1397] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:29:32.643462 2640 ts_tablet_manager.cc:1397] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 18:29:32.644481 2641 tablet_bootstrap.cc:492] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: No bootstrap required, opened a new log
I20260620 18:29:32.644737 2641 ts_tablet_manager.cc:1397] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 18:29:32.652937 2641 raft_consensus.cc:348] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.653327 2641 raft_consensus.cc:374] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.653442 2641 raft_consensus.cc:729] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999831788cd8412696c34774394578c6, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.654105 2641 consensus_queue.cc:260] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.653972 2640 raft_consensus.cc:348] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.653972 2642 raft_consensus.cc:348] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.654382 2642 raft_consensus.cc:374] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.654382 2640 raft_consensus.cc:374] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.654477 2640 raft_consensus.cc:729] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.654489 2642 raft_consensus.cc:729] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.655082 2640 consensus_queue.cc:260] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.655205 2642 consensus_queue.cc:260] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.657073 2621 heartbeater.cc:503] Master 127.0.97.62:46791 was elected leader, sending a full tablet report...
I20260620 18:29:32.657760 2641 ts_tablet_manager.cc:1428] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:29:32.661131 2642 ts_tablet_manager.cc:1428] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Time spent starting tablet: real 0.018s user 0.003s sys 0.011s
I20260620 18:29:32.662000 2482 heartbeater.cc:503] Master 127.0.97.62:46791 was elected leader, sending a full tablet report...
I20260620 18:29:32.662598 2640 ts_tablet_manager.cc:1428] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
W20260620 18:29:32.734390 2344 tablet.cc:2369] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:32.755872 2483 tablet.cc:2369] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:32.761840 2622 tablet.cc:2369] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:32.771818 2647 raft_consensus.cc:482] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:32.772166 2647 raft_consensus.cc:504] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.774041 2647 leader_election.cc:288] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:32.781140 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.774866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45055 (local address 127.0.97.1:37929)
0620 18:29:32.775102 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.775111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.775180 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:29:32.775443 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.775447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.775568 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.775676 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:32.776303 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.777220 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.779236 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.780188 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.780286 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.780363 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.780755 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.780879 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:29:32.781184 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.774361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37929 (local address 127.0.97.2:45055)
0620 18:29:32.775082 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.775103 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.775250 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.775702 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.775708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.775738 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:32.776138 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.776150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.777440 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.777446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.779083 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.779097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.780404 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.780412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.780588 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.780627 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.780716 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.780864 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":794,"threads_started":1}
I20260620 18:29:32.781497 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.774898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40277 (local address 127.0.97.3:42137)
0620 18:29:32.775077 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.775083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.776008 (+ 925us) server_negotiation.cc:407] Connection header received
0620 18:29:32.776132 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.776137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.776215 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.776299 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:32.776899 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.777919 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.779720 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.780590 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.780714 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.780809 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.781148 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.781289 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:29:32.781818 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.774404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42137 (local address 127.0.97.2:40277)
0620 18:29:32.775903 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.775917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.776027 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.776326 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.776331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.776357 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:32.776742 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.776754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.778052 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.778056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.779584 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.779598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.780825 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.780831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.780986 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.781018 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.781108 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.781243 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1186,"threads_started":1}
I20260620 18:29:32.782728 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037f899e8f914b7bbaa85c1b993b82bf" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5" is_pre_election: true
I20260620 18:29:32.782752 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037f899e8f914b7bbaa85c1b993b82bf" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6" is_pre_election: true
I20260620 18:29:32.783281 2298 raft_consensus.cc:2454] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 0.
I20260620 18:29:32.783281 2571 raft_consensus.cc:2454] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 0.
I20260620 18:29:32.784039 2363 leader_election.cc:302] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, 999831788cd8412696c34774394578c6, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:32.784397 2647 raft_consensus.cc:2781] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:32.784541 2647 raft_consensus.cc:482] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:32.784651 2647 raft_consensus.cc:3037] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:32.786744 2647 raft_consensus.cc:504] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.787338 2647 leader_election.cc:288] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 election: Requested vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:32.787770 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037f899e8f914b7bbaa85c1b993b82bf" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5"
I20260620 18:29:32.787957 2298 raft_consensus.cc:3037] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:32.788040 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037f899e8f914b7bbaa85c1b993b82bf" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6"
I20260620 18:29:32.788290 2571 raft_consensus.cc:3037] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:32.790151 2298 raft_consensus.cc:2454] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 1.
I20260620 18:29:32.790416 2571 raft_consensus.cc:2454] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 1.
I20260620 18:29:32.790570 2363 leader_election.cc:302] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:32.790853 2647 raft_consensus.cc:2781] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:32.791283 2647 raft_consensus.cc:686] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 1 LEADER]: Becoming Leader. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Running, Role: LEADER
I20260620 18:29:32.791790 2647 consensus_queue.cc:237] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } }
I20260620 18:29:32.796627 2134 catalog_manager.cc:5495] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74c3f0649eb4012930065159a800b7a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } health_report { overall_health: HEALTHY } } }
Jun 20 18:29:32 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:32.878917 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.865928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46791 (local address 127.0.0.1:37060)
0620 18:29:32.866452 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:32.866467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:32.866641 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:32.866941 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:32.866944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:32.867755 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:32.867998 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.868006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.869152 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.869156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.870159 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:32.870170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.870863 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.870867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:32.870980 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.871674 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:32.871690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:32.873801 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:32.876799 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.876808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.876814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.877218 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.877543 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:32.877547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:32.877549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:32.877686 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:32.878135 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:32.878142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:32.878280 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:29:32.878418 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:29:32.878610 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":300,"threads_started":1}
I20260620 18:29:32.878960 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:32.865915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.97.62:46791)
0620 18:29:32.866102 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:32.866107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:32.866550 (+ 443us) server_negotiation.cc:407] Connection header received
0620 18:29:32.866741 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:32.866745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:32.866802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:32.866894 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:32.868139 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.869030 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.870284 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:32.870706 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:32.870802 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:32.873962 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:32.873966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:32.873970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:32.873996 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:32.876672 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.877317 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.877321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.877323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.877444 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:32.877775 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:32.877778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:32.877780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:32.878007 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:32.878110 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:32.878490 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 18:29:32.878720 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":55}
I20260620 18:29:32.883975 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:29:32.885010 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:32.891583 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4faabf20b64b0399991073caf1de24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31dafc3a22ef467da44f91d44cab4ffc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:32.891808 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4faabf20b64b0399991073caf1de24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31dafc3a22ef467da44f91d44cab4ffc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:32.891808 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4faabf20b64b0399991073caf1de24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31dafc3a22ef467da44f91d44cab4ffc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:29:32.892616 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4faabf20b64b0399991073caf1de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.892752 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4faabf20b64b0399991073caf1de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.892752 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4faabf20b64b0399991073caf1de24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:32.898555 2642 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: Bootstrap starting.
I20260620 18:29:32.900091 2642 tablet_bootstrap.cc:654] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.900574 2640 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: Bootstrap starting.
I20260620 18:29:32.900574 2641 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: Bootstrap starting.
I20260620 18:29:32.902285 2641 tablet_bootstrap.cc:654] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.902283 2640 tablet_bootstrap.cc:654] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:32.904345 2642 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: No bootstrap required, opened a new log
I20260620 18:29:32.904660 2642 ts_tablet_manager.cc:1397] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 18:29:32.905233 2641 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: No bootstrap required, opened a new log
I20260620 18:29:32.905390 2641 ts_tablet_manager.cc:1397] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260620 18:29:32.905771 2640 tablet_bootstrap.cc:492] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: No bootstrap required, opened a new log
I20260620 18:29:32.905668 2642 raft_consensus.cc:348] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.905913 2640 ts_tablet_manager.cc:1397] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 18:29:32.905918 2642 raft_consensus.cc:374] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.906026 2642 raft_consensus.cc:729] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.906088 2641 raft_consensus.cc:348] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.906371 2641 raft_consensus.cc:374] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.906311 2642 consensus_queue.cc:260] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.906467 2641 raft_consensus.cc:729] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999831788cd8412696c34774394578c6, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.906701 2640 raft_consensus.cc:348] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.906723 2641 consensus_queue.cc:260] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.906939 2640 raft_consensus.cc:374] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:32.907035 2640 raft_consensus.cc:729] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Initialized, Role: FOLLOWER
I20260620 18:29:32.906838 2648 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:32.907183 2648 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:32.907335 2641 ts_tablet_manager.cc:1428] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:29:32.907285 2640 consensus_queue.cc:260] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:32.907723 2647 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:32.907887 2640 ts_tablet_manager.cc:1428] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:29:32.908002 2647 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:32.907214 2646 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:32.908171 2646 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:32.908327 2642 ts_tablet_manager.cc:1428] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:29:33.034075 2647 raft_consensus.cc:482] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:33.034319 2647 raft_consensus.cc:504] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.035090 2647 leader_election.cc:288] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:33.035669 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4faabf20b64b0399991073caf1de24" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6" is_pre_election: true
I20260620 18:29:33.035642 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4faabf20b64b0399991073caf1de24" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5" is_pre_election: true
I20260620 18:29:33.035900 2571 raft_consensus.cc:2454] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 0.
I20260620 18:29:33.035964 2298 raft_consensus.cc:2454] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 0.
I20260620 18:29:33.036357 2365 leader_election.cc:302] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 999831788cd8412696c34774394578c6, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:33.036657 2647 raft_consensus.cc:2781] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:33.036758 2647 raft_consensus.cc:482] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:33.036911 2647 raft_consensus.cc:3037] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.038854 2647 raft_consensus.cc:504] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.039388 2647 leader_election.cc:288] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 election: Requested vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:33.039834 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4faabf20b64b0399991073caf1de24" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5"
I20260620 18:29:33.039836 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4faabf20b64b0399991073caf1de24" candidate_uuid: "f74c3f0649eb4012930065159a800b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6"
I20260620 18:29:33.040051 2298 raft_consensus.cc:3037] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.040058 2571 raft_consensus.cc:3037] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.042088 2571 raft_consensus.cc:2454] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 1.
I20260620 18:29:33.042392 2298 raft_consensus.cc:2454] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c3f0649eb4012930065159a800b7a in term 1.
I20260620 18:29:33.042462 2365 leader_election.cc:302] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 999831788cd8412696c34774394578c6, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:33.042755 2647 raft_consensus.cc:2781] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:33.042958 2647 raft_consensus.cc:686] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [term 1 LEADER]: Becoming Leader. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Running, Role: LEADER
I20260620 18:29:33.043193 2647 consensus_queue.cc:237] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.044786 2651 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: TxnStatusTablet state changed. Reason: New leader f74c3f0649eb4012930065159a800b7a. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.045174 2651 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:33.045913 2663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:29:33.046250 2663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:29:33.046211 2135 catalog_manager.cc:5495] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74c3f0649eb4012930065159a800b7a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } health_report { overall_health: UNKNOWN } } }
I20260620 18:29:33.098999 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.092403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.97.2:33801)
0620 18:29:33.092609 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.092614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.093020 (+ 406us) server_negotiation.cc:407] Connection header received
0620 18:29:33.093343 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.093350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.093458 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.093648 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.094017 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.094752 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.096638 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.097058 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.097147 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.097740 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.098442 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.098558 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.098689 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.098756 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:29:33.099128 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.092565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.0.1:56526)
0620 18:29:33.092931 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.092952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.093106 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.093666 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.093671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.093685 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:33.093875 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.093886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.094885 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.094891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.096509 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.096524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.097211 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.097226 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.097451 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.097526 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.098552 (+ 1026us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.098573 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.098665 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.098817 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260620 18:29:33.106546 2298 raft_consensus.cc:1261] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Refusing update from remote peer f74c3f0649eb4012930065159a800b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.106679 2571 raft_consensus.cc:1261] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Refusing update from remote peer f74c3f0649eb4012930065159a800b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.107596 2647 consensus_queue.cc:1035] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.107968 2651 consensus_queue.cc:1035] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.115710 2648 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: TxnStatusTablet state changed. Reason: New leader f74c3f0649eb4012930065159a800b7a. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.116005 2648 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:33.118314 2666 mvcc.cc:205] Tried to move back new op lower bound from 7298990789034897408 to 7298990788787183616. Current Snapshot: MvccSnapshot[applied={T|T < 7298990789034897408}]
I20260620 18:29:33.122579 2648 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.122808 2648 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 3f461ea159ba4d2eac934c3b417f68a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:33.123554 2647 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.123858 2647 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:33.129597 2651 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.129845 2651 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P f74c3f0649eb4012930065159a800b7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:29:33.130066 2646 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: TxnStatusTablet state changed. Reason: New leader f74c3f0649eb4012930065159a800b7a. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.130307 2646 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:33.132246 2668 mvcc.cc:205] Tried to move back new op lower bound from 7298990789034897408 to 7298990788787183616. Current Snapshot: MvccSnapshot[applied={T|T < 7298990789034897408}]
I20260620 18:29:33.133124 2646 tablet_replica.cc:439] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f74c3f0649eb4012930065159a800b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } } }
I20260620 18:29:33.133375 2646 tablet_replica.cc:442] T 2b4faabf20b64b0399991073caf1de24 P 999831788cd8412696c34774394578c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:29:33.142279 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.135546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.97.2:33801)
0620 18:29:33.135712 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.135718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.136271 (+ 553us) server_negotiation.cc:407] Connection header received
0620 18:29:33.136548 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.136553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.136644 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.136705 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.137371 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.138133 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.140197 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.140634 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.140738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.141229 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.141430 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.141565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.141778 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.141875 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:29:33.161124 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.154071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51489 (local address 127.0.97.2:33801)
0620 18:29:33.154271 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.154278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.154903 (+ 625us) server_negotiation.cc:407] Connection header received
0620 18:29:33.155060 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.155066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.155157 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.155267 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.155820 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.157080 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.158655 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.159459 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.159626 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.160123 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.160267 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.160390 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.160538 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.160651 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":55}
I20260620 18:29:33.161187 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.154102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.97.2:51489)
0620 18:29:33.154793 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.154811 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.154933 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.155281 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.155284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.155293 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:33.155676 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.155685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.157292 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.157296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.158515 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.158527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.159700 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.159705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.159832 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.159898 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.160397 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.160407 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.160501 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.160618 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":3,"thread_start_us":365,"threads_started":1}
I20260620 18:29:33.170859 2298 raft_consensus.cc:1261] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Refusing update from remote peer f74c3f0649eb4012930065159a800b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.170863 2571 raft_consensus.cc:1261] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Refusing update from remote peer f74c3f0649eb4012930065159a800b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.171397 2651 consensus_queue.cc:1035] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.171763 2647 consensus_queue.cc:1035] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.235981 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:33.236269 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:33.239900 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Sending DeleteTablet for 3 replicas of tablet 037f899e8f914b7bbaa85c1b993b82bf
I20260620 18:29:33.241014 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260620 18:29:33.241171 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57664
I20260620 18:29:33.242086 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:29:33.242169 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 037f899e8f914b7bbaa85c1b993b82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56510
W20260620 18:29:33.242728 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:33.249676 2272 tablet_service.cc:1467] Processing CreateTablet for tablet af21eaf061304c8eb8834e1dc93db38f (DEFAULT_TABLE table=test-table [id=bf3d60adea834346915cac6b1481aba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:33.250082 2411 tablet_service.cc:1467] Processing CreateTablet for tablet af21eaf061304c8eb8834e1dc93db38f (DEFAULT_TABLE table=test-table [id=bf3d60adea834346915cac6b1481aba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:33.250278 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af21eaf061304c8eb8834e1dc93db38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:33.250695 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af21eaf061304c8eb8834e1dc93db38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:33.252276 2551 tablet_service.cc:1467] Processing CreateTablet for tablet af21eaf061304c8eb8834e1dc93db38f (DEFAULT_TABLE table=test-table [id=bf3d60adea834346915cac6b1481aba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:33.252837 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af21eaf061304c8eb8834e1dc93db38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:33.256738 2642 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Bootstrap starting.
I20260620 18:29:33.257637 2683 tablet_replica.cc:330] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: stopping tablet replica
I20260620 18:29:33.258158 2683 raft_consensus.cc:2229] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:33.258535 2642 tablet_bootstrap.cc:654] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:33.258559 2683 raft_consensus.cc:2258] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:33.258945 2641 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Bootstrap starting.
I20260620 18:29:33.260672 2641 tablet_bootstrap.cc:654] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:33.260643 2683 ts_tablet_manager.cc:1905] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:33.260710 2642 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: No bootstrap required, opened a new log
I20260620 18:29:33.261332 2642 ts_tablet_manager.cc:1397] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 18:29:33.262135 2642 raft_consensus.cc:348] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.262487 2642 raft_consensus.cc:374] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:33.262616 2642 raft_consensus.cc:729] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Initialized, Role: FOLLOWER
I20260620 18:29:33.262951 2642 consensus_queue.cc:260] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.266376 2642 ts_tablet_manager.cc:1428] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 18:29:33.269116 2683 ts_tablet_manager.cc:1918] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:33.269272 2683 log.cc:1196] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/037f899e8f914b7bbaa85c1b993b82bf
I20260620 18:29:33.269807 2683 ts_tablet_manager.cc:1939] T 037f899e8f914b7bbaa85c1b993b82bf P 3f461ea159ba4d2eac934c3b417f68a5: Deleting consensus metadata
I20260620 18:29:33.271587 2120 catalog_manager.cc:4841] TS 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929): tablet 037f899e8f914b7bbaa85c1b993b82bf (table test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]) successfully deleted
I20260620 18:29:33.276801 2641 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: No bootstrap required, opened a new log
I20260620 18:29:33.276978 2641 ts_tablet_manager.cc:1397] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.006s
I20260620 18:29:33.277753 2641 raft_consensus.cc:348] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.278002 2641 raft_consensus.cc:374] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:33.278082 2641 raft_consensus.cc:729] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999831788cd8412696c34774394578c6, State: Initialized, Role: FOLLOWER
I20260620 18:29:33.278312 2641 consensus_queue.cc:260] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.278546 2640 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Bootstrap starting.
I20260620 18:29:33.279739 2641 ts_tablet_manager.cc:1428] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:29:33.280723 2640 tablet_bootstrap.cc:654] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:33.282194 2682 tablet_replica.cc:330] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: stopping tablet replica
I20260620 18:29:33.282600 2682 raft_consensus.cc:2229] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:33.282964 2682 raft_consensus.cc:2258] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:33.285073 2682 ts_tablet_manager.cc:1905] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:33.288795 2684 tablet_replica.cc:330] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: stopping tablet replica
I20260620 18:29:33.289305 2684 raft_consensus.cc:2229] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:33.289958 2684 raft_consensus.cc:2258] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:33.290372 2640 tablet_bootstrap.cc:492] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: No bootstrap required, opened a new log
I20260620 18:29:33.290566 2640 ts_tablet_manager.cc:1397] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260620 18:29:33.291332 2640 raft_consensus.cc:348] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.291603 2640 raft_consensus.cc:374] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:33.291694 2640 raft_consensus.cc:729] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Initialized, Role: FOLLOWER
I20260620 18:29:33.292068 2684 ts_tablet_manager.cc:1905] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:33.291996 2640 consensus_queue.cc:260] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.293098 2682 ts_tablet_manager.cc:1918] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:33.293253 2682 log.cc:1196] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/037f899e8f914b7bbaa85c1b993b82bf
I20260620 18:29:33.293709 2682 ts_tablet_manager.cc:1939] T 037f899e8f914b7bbaa85c1b993b82bf P 999831788cd8412696c34774394578c6: Deleting consensus metadata
I20260620 18:29:33.295075 2640 ts_tablet_manager.cc:1428] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:29:33.295300 2122 catalog_manager.cc:4841] TS 999831788cd8412696c34774394578c6 (127.0.97.3:42137): tablet 037f899e8f914b7bbaa85c1b993b82bf (table test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]) successfully deleted
I20260620 18:29:33.298610 2684 ts_tablet_manager.cc:1918] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:33.298796 2684 log.cc:1196] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/037f899e8f914b7bbaa85c1b993b82bf
I20260620 18:29:33.299274 2684 ts_tablet_manager.cc:1939] T 037f899e8f914b7bbaa85c1b993b82bf P f74c3f0649eb4012930065159a800b7a: Deleting consensus metadata
I20260620 18:29:33.300568 2122 catalog_manager.cc:4841] TS f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801): tablet 037f899e8f914b7bbaa85c1b993b82bf (table test-table [id=5874c158b7f942aaa8fa3b23567b3b5b]) successfully deleted
I20260620 18:29:33.366945 2646 raft_consensus.cc:482] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:33.367149 2646 raft_consensus.cc:504] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.368851 2646 leader_election.cc:288] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801)
I20260620 18:29:33.377214 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.369394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59123 (local address 127.0.97.1:37929)
0620 18:29:33.370972 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.370978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.370999 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:29:33.371069 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.371075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.371160 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.371280 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:33.371844 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.372983 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.375124 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.376189 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.376312 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.376432 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.376736 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.376874 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":1314,"threads_started":1}
I20260620 18:29:33.377254 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.369018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37929 (local address 127.0.97.3:59123)
0620 18:29:33.370220 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.370246 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.370463 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.371301 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.371306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.371342 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:33.371638 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.371650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.373143 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.373148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.374976 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.374988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.376358 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.376364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.376555 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.376593 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.376692 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.376836 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":29,"thread_start_us":845,"threads_started":1}
I20260620 18:29:33.378099 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af21eaf061304c8eb8834e1dc93db38f" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5" is_pre_election: true
I20260620 18:29:33.378232 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.369468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37047 (local address 127.0.97.2:33801)
0620 18:29:33.369711 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.369720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.371306 (+ 1586us) server_negotiation.cc:407] Connection header received
0620 18:29:33.371633 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.371639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.371724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.371813 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:33.372349 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.373145 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.375713 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.377039 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.377195 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.377295 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.377839 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.377963 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:29:33.378424 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.369135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.97.3:37047)
0620 18:29:33.371202 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.371223 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.371421 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.371940 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.371943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.371970 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:33.372199 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.372205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.373311 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.373316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.375546 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.375561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.377425 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.377432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.377622 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.377657 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.377787 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.377908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"mutex_wait_us":39,"thread_start_us":1785,"threads_started":1}
I20260620 18:29:33.378392 2297 raft_consensus.cc:2454] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 0.
I20260620 18:29:33.378849 2502 leader_election.cc:302] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, 999831788cd8412696c34774394578c6; no voters:
I20260620 18:29:33.379235 2646 raft_consensus.cc:2781] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:33.379382 2646 raft_consensus.cc:482] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:33.379287 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af21eaf061304c8eb8834e1dc93db38f" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a" is_pre_election: true
I20260620 18:29:33.379488 2646 raft_consensus.cc:3037] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.379698 2437 raft_consensus.cc:2454] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 0.
I20260620 18:29:33.381657 2646 raft_consensus.cc:504] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.382233 2646 leader_election.cc:288] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 election: Requested vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801)
I20260620 18:29:33.382668 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af21eaf061304c8eb8834e1dc93db38f" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5"
I20260620 18:29:33.382700 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af21eaf061304c8eb8834e1dc93db38f" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a"
I20260620 18:29:33.382849 2297 raft_consensus.cc:3037] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.382925 2437 raft_consensus.cc:3037] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:33.384943 2297 raft_consensus.cc:2454] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 1.
I20260620 18:29:33.385411 2502 leader_election.cc:302] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, 999831788cd8412696c34774394578c6; no voters:
I20260620 18:29:33.385771 2646 raft_consensus.cc:2781] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:33.386241 2437 raft_consensus.cc:2454] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 1.
I20260620 18:29:33.386816 2646 raft_consensus.cc:686] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 1 LEADER]: Becoming Leader. State: Replica: 999831788cd8412696c34774394578c6, State: Running, Role: LEADER
I20260620 18:29:33.387195 2646 consensus_queue.cc:237] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:33.390717 2134 catalog_manager.cc:5495] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 999831788cd8412696c34774394578c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "999831788cd8412696c34774394578c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } health_report { overall_health: HEALTHY } } }
I20260620 18:29:33.435882 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.430401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.97.3:42137)
0620 18:29:33.430851 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.430856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.430876 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:29:33.430963 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.430966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.431032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.431202 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.431618 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.432350 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.433660 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.434092 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.434224 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.434678 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.435246 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.435420 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.435471 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.435570 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260620 18:29:33.452636 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.445977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.97.3:49951)
0620 18:29:33.446765 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.446796 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.446935 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.447339 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.447343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.447356 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:33.447754 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.447764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.448827 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.448833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.450441 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.450454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.451240 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.451247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.451422 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.451498 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.451997 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.452009 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.452095 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.452235 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":375,"threads_started":1}
I20260620 18:29:33.453266 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.446038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49951 (local address 127.0.97.2:33801)
0620 18:29:33.446257 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.446268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.446908 (+ 640us) server_negotiation.cc:407] Connection header received
0620 18:29:33.447066 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.447070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.447213 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.447300 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.447921 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.448684 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.450580 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.451063 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.451164 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.451695 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.451855 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.451944 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.452961 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.453029 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:29:33.463255 2297 raft_consensus.cc:1261] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Refusing update from remote peer 999831788cd8412696c34774394578c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.463482 2437 raft_consensus.cc:1261] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Refusing update from remote peer 999831788cd8412696c34774394578c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:33.464002 2691 consensus_queue.cc:1035] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.464432 2646 consensus_queue.cc:1035] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:33.494442 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.487485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53527 (local address 127.0.97.3:42137)
0620 18:29:33.488689 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:33.488695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:33.488712 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:29:33.488770 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:33.488774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:33.488862 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:33.488951 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:33.489438 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.490511 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.492501 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.493121 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.493227 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.493728 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.493846 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.493930 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:33.494072 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:29:33.494149 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096}
I20260620 18:29:33.494519 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:33.487412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42137 (local address 127.0.97.2:53527)
0620 18:29:33.487680 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:33.487700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:33.487810 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:33.489112 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:33.489117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:33.489128 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:33.489305 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.489313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.490653 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.490660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.492338 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:33.492353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:33.493341 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:33.493347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:33.493518 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:33.493564 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:33.493950 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:33.493961 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:29:33.494040 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:29:33.494208 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 18:29:33.495693 2668 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:29:35.109997 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:35.110205 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:35.114064 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Sending DeleteTablet for 3 replicas of tablet af21eaf061304c8eb8834e1dc93db38f
I20260620 18:29:35.114777 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet af21eaf061304c8eb8834e1dc93db38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56510
I20260620 18:29:35.114876 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet af21eaf061304c8eb8834e1dc93db38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57664
I20260620 18:29:35.114902 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet af21eaf061304c8eb8834e1dc93db38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260620 18:29:35.126935 2722 tablet_replica.cc:330] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: stopping tablet replica
I20260620 18:29:35.127216 2724 tablet_replica.cc:330] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: stopping tablet replica
I20260620 18:29:35.127393 2723 tablet_replica.cc:330] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: stopping tablet replica
I20260620 18:29:35.127427 2722 raft_consensus.cc:2229] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:35.127579 2724 raft_consensus.cc:2229] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:35.127621 2723 raft_consensus.cc:2229] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:35.127772 2722 raft_consensus.cc:2258] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.127898 2724 raft_consensus.cc:2258] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.128193 2723 raft_consensus.cc:2258] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.128937 2722 ts_tablet_manager.cc:1905] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.128961 2724 ts_tablet_manager.cc:1905] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.129074 2723 ts_tablet_manager.cc:1905] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.135538 2723 ts_tablet_manager.cc:1918] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:35.135602 2724 ts_tablet_manager.cc:1918] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:35.135717 2724 log.cc:1196] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/af21eaf061304c8eb8834e1dc93db38f
I20260620 18:29:35.135727 2723 log.cc:1196] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/af21eaf061304c8eb8834e1dc93db38f
I20260620 18:29:35.136157 2723 ts_tablet_manager.cc:1939] T af21eaf061304c8eb8834e1dc93db38f P 999831788cd8412696c34774394578c6: Deleting consensus metadata
I20260620 18:29:35.136258 2724 ts_tablet_manager.cc:1939] T af21eaf061304c8eb8834e1dc93db38f P 3f461ea159ba4d2eac934c3b417f68a5: Deleting consensus metadata
I20260620 18:29:35.136751 2722 ts_tablet_manager.cc:1918] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:35.136955 2722 log.cc:1196] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/af21eaf061304c8eb8834e1dc93db38f
I20260620 18:29:35.137270 2122 catalog_manager.cc:4841] TS 999831788cd8412696c34774394578c6 (127.0.97.3:42137): tablet af21eaf061304c8eb8834e1dc93db38f (table test-table [id=bf3d60adea834346915cac6b1481aba7]) successfully deleted
I20260620 18:29:35.137256 2120 catalog_manager.cc:4841] TS 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929): tablet af21eaf061304c8eb8834e1dc93db38f (table test-table [id=bf3d60adea834346915cac6b1481aba7]) successfully deleted
I20260620 18:29:35.137615 2722 ts_tablet_manager.cc:1939] T af21eaf061304c8eb8834e1dc93db38f P f74c3f0649eb4012930065159a800b7a: Deleting consensus metadata
I20260620 18:29:35.138526 2122 catalog_manager.cc:4841] TS f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801): tablet af21eaf061304c8eb8834e1dc93db38f (table test-table [id=bf3d60adea834346915cac6b1481aba7]) successfully deleted
Jun 20 18:29:35 dist-test-slave-mcnm krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:29:35 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:35.173516 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.158343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.0.97.62:46791)
0620 18:29:35.158904 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.158912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.158988 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:29:35.159231 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.159235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.159315 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.159435 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:35.160770 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.161614 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.163315 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.163757 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.163902 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.167904 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:35.167909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:35.167912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:35.167955 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:35.170862 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:35.171579 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:35.171583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:35.171585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:35.171700 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:35.172142 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:35.172146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:35.172148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:35.172436 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:35.172657 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.172933 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.173204 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":271,"threads_started":1}
I20260620 18:29:35.177124 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:35.177681 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:35.183831 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 8939969c7276465c9aa342271065c15b (DEFAULT_TABLE table=test-table [id=651acc8a859543e79d33b343dd5cdeeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.183934 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 8939969c7276465c9aa342271065c15b (DEFAULT_TABLE table=test-table [id=651acc8a859543e79d33b343dd5cdeeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.184334 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8939969c7276465c9aa342271065c15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.184412 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8939969c7276465c9aa342271065c15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.189301 2733 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Bootstrap starting.
I20260620 18:29:35.189626 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 8939969c7276465c9aa342271065c15b (DEFAULT_TABLE table=test-table [id=651acc8a859543e79d33b343dd5cdeeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.189918 2734 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Bootstrap starting.
I20260620 18:29:35.190058 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8939969c7276465c9aa342271065c15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.191443 2733 tablet_bootstrap.cc:654] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.191582 2734 tablet_bootstrap.cc:654] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.193553 2733 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: No bootstrap required, opened a new log
I20260620 18:29:35.193693 2733 ts_tablet_manager.cc:1397] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:29:35.193768 2734 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: No bootstrap required, opened a new log
I20260620 18:29:35.193897 2734 ts_tablet_manager.cc:1397] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 18:29:35.194470 2734 raft_consensus.cc:348] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.194418 2733 raft_consensus.cc:348] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.194677 2734 raft_consensus.cc:374] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.194792 2734 raft_consensus.cc:729] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.195036 2734 consensus_queue.cc:260] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.195111 2733 raft_consensus.cc:374] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.195223 2733 raft_consensus.cc:729] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.195457 2733 consensus_queue.cc:260] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.196025 2733 ts_tablet_manager.cc:1428] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:29:35.197022 2738 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Bootstrap starting.
I20260620 18:29:35.197695 2734 ts_tablet_manager.cc:1428] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260620 18:29:35.199337 2738 tablet_bootstrap.cc:654] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.201398 2738 tablet_bootstrap.cc:492] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: No bootstrap required, opened a new log
I20260620 18:29:35.201543 2738 ts_tablet_manager.cc:1397] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:29:35.202126 2738 raft_consensus.cc:348] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.202327 2738 raft_consensus.cc:374] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.202426 2738 raft_consensus.cc:729] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999831788cd8412696c34774394578c6, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.202692 2738 consensus_queue.cc:260] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.203199 2738 ts_tablet_manager.cc:1428] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:29:35.240432 2737 raft_consensus.cc:482] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:35.240693 2737 raft_consensus.cc:504] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.242213 2737 leader_election.cc:288] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:35.250180 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.242715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51181 (local address 127.0.97.2:33801)
0620 18:29:35.244354 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.244361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.244385 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:29:35.244453 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.244456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.244508 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.244619 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:35.245176 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.246257 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.248256 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.249205 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.249362 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.249463 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.249768 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.249890 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":894,"threads_started":1}
I20260620 18:29:35.250195 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.243613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40045 (local address 127.0.97.3:42137)
0620 18:29:35.244522 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.244527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.244544 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:29:35.244596 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.244599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.244650 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.244772 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:35.245352 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.246172 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.248206 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.249174 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.249271 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.249409 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.249753 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.249881 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":673,"threads_started":1}
I20260620 18:29:35.250283 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.242673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.97.1:51181)
0620 18:29:35.243210 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:35.243228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:35.243367 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:35.244637 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:35.244641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:35.244667 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:35.244991 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.245001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.246401 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.246406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.248112 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.248145 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.249426 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.249434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.249603 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.249625 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:29:35.249728 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:29:35.249856 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":31,"thread_start_us":301,"threads_started":1}
I20260620 18:29:35.250751 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.243455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42137 (local address 127.0.97.1:40045)
0620 18:29:35.244124 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:35.244138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:35.244278 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:35.244788 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:35.244792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:35.244823 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:35.245199 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.245207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.246296 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.246304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.248049 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.248064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.249364 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.249370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.249555 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.249610 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:29:35.249689 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:29:35.249836 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":46,"thread_start_us":389,"threads_started":1}
I20260620 18:29:35.250854 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8939969c7276465c9aa342271065c15b" candidate_uuid: "3f461ea159ba4d2eac934c3b417f68a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a" is_pre_election: true
I20260620 18:29:35.251077 2437 raft_consensus.cc:2454] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f461ea159ba4d2eac934c3b417f68a5 in term 0.
I20260620 18:29:35.251560 2225 leader_election.cc:302] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:35.251804 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8939969c7276465c9aa342271065c15b" candidate_uuid: "3f461ea159ba4d2eac934c3b417f68a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6" is_pre_election: true
I20260620 18:29:35.251991 2737 raft_consensus.cc:2781] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:35.252043 2571 raft_consensus.cc:2454] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f461ea159ba4d2eac934c3b417f68a5 in term 0.
I20260620 18:29:35.252127 2737 raft_consensus.cc:482] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:35.252219 2737 raft_consensus.cc:3037] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.254765 2737 raft_consensus.cc:504] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.255342 2737 leader_election.cc:288] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [CANDIDATE]: Term 1 election: Requested vote from peers f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801), 999831788cd8412696c34774394578c6 (127.0.97.3:42137)
I20260620 18:29:35.255733 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8939969c7276465c9aa342271065c15b" candidate_uuid: "3f461ea159ba4d2eac934c3b417f68a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a"
I20260620 18:29:35.255957 2437 raft_consensus.cc:3037] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.255931 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8939969c7276465c9aa342271065c15b" candidate_uuid: "3f461ea159ba4d2eac934c3b417f68a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999831788cd8412696c34774394578c6"
I20260620 18:29:35.256158 2571 raft_consensus.cc:3037] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.258098 2437 raft_consensus.cc:2454] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f461ea159ba4d2eac934c3b417f68a5 in term 1.
I20260620 18:29:35.258390 2571 raft_consensus.cc:2454] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f461ea159ba4d2eac934c3b417f68a5 in term 1.
I20260620 18:29:35.258448 2225 leader_election.cc:302] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:35.258863 2737 raft_consensus.cc:2781] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:35.259258 2737 raft_consensus.cc:686] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Running, Role: LEADER
I20260620 18:29:35.259665 2737 consensus_queue.cc:237] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.262889 2136 catalog_manager.cc:5495] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3f461ea159ba4d2eac934c3b417f68a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } health_report { overall_health: UNKNOWN } } }
I20260620 18:29:35.303074 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.297017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.0.97.1:37929)
0620 18:29:35.297578 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.297582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.297599 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:29:35.297675 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.297678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.297760 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.297942 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:35.298350 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.299578 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.300878 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.301294 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.301421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.301872 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.302465 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.302629 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.302679 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.302750 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":326,"threads_started":1}
I20260620 18:29:35.318360 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.312574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36467 (local address 127.0.97.2:33801)
0620 18:29:35.312751 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.312755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.313301 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:29:35.313375 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.313379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.313445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.313531 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:35.314011 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.314789 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.316215 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.316780 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.316941 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.317610 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.317778 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.317868 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.317986 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.318047 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:29:35.318727 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.312495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33801 (local address 127.0.97.1:36467)
0620 18:29:35.313156 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:35.313172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:35.313279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:35.313576 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:35.313580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:35.313590 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:35.313876 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.313883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.314932 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.314936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.316073 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.316084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.317026 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.317032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.317307 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.317406 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.317880 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.317890 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:29:35.317956 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:29:35.318051 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":359,"threads_started":1}
I20260620 18:29:35.326231 2571 raft_consensus.cc:1261] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Refusing update from remote peer 3f461ea159ba4d2eac934c3b417f68a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:35.326411 2437 raft_consensus.cc:1261] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Refusing update from remote peer 3f461ea159ba4d2eac934c3b417f68a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:35.326988 2744 consensus_queue.cc:1035] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:35.327404 2737 consensus_queue.cc:1035] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:35.357991 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.351960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36515 (local address 127.0.97.1:37929)
0620 18:29:35.352118 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.352123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.352498 (+ 375us) server_negotiation.cc:407] Connection header received
0620 18:29:35.352612 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.352615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.352671 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.352756 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:35.353127 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.354078 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.356079 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.356560 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.356718 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.357270 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.357362 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.357461 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.357604 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.357674 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:29:35.358098 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.351860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37929 (local address 127.0.97.2:36515)
0620 18:29:35.352416 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:35.352431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:35.352532 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:35.352780 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:35.352783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:35.352791 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:29:35.352985 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.352992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.354203 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.354209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.355936 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:35.355950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.356846 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.356853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:35.357032 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.357103 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.357466 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.357479 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:29:35.357575 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:29:35.357703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
I20260620 18:29:35.387671 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:35.387928 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:35.390813 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Sending DeleteTablet for 3 replicas of tablet 8939969c7276465c9aa342271065c15b
I20260620 18:29:35.391780 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8939969c7276465c9aa342271065c15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57664
I20260620 18:29:35.391777 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 8939969c7276465c9aa342271065c15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260620 18:29:35.392066 2723 tablet_replica.cc:330] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: stopping tablet replica
I20260620 18:29:35.392069 2724 tablet_replica.cc:330] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: stopping tablet replica
I20260620 18:29:35.392267 2723 raft_consensus.cc:2229] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:35.392298 2724 raft_consensus.cc:2229] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:35.392482 2723 raft_consensus.cc:2258] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.392778 2724 raft_consensus.cc:2258] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.393509 2723 ts_tablet_manager.cc:1905] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.393957 2724 ts_tablet_manager.cc:1905] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.397512 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 8939969c7276465c9aa342271065c15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56510
I20260620 18:29:35.397884 2722 tablet_replica.cc:330] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: stopping tablet replica
I20260620 18:29:35.398092 2722 raft_consensus.cc:2229] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:35.398304 2722 raft_consensus.cc:2258] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:35.398547 2723 ts_tablet_manager.cc:1918] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:35.398706 2723 log.cc:1196] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/8939969c7276465c9aa342271065c15b
I20260620 18:29:35.399081 2723 ts_tablet_manager.cc:1939] T 8939969c7276465c9aa342271065c15b P 999831788cd8412696c34774394578c6: Deleting consensus metadata
I20260620 18:29:35.399552 2722 ts_tablet_manager.cc:1905] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:35.400132 2122 catalog_manager.cc:4841] TS 999831788cd8412696c34774394578c6 (127.0.97.3:42137): tablet 8939969c7276465c9aa342271065c15b (table test-table [id=651acc8a859543e79d33b343dd5cdeeb]) successfully deleted
I20260620 18:29:35.400969 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:29:35.401818 2724 ts_tablet_manager.cc:1918] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:35.401983 2724 log.cc:1196] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/8939969c7276465c9aa342271065c15b
W20260620 18:29:35.402292 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:35.402402 2724 ts_tablet_manager.cc:1939] T 8939969c7276465c9aa342271065c15b P 3f461ea159ba4d2eac934c3b417f68a5: Deleting consensus metadata
I20260620 18:29:35.403532 2120 catalog_manager.cc:4841] TS 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929): tablet 8939969c7276465c9aa342271065c15b (table test-table [id=651acc8a859543e79d33b343dd5cdeeb]) successfully deleted
I20260620 18:29:35.405382 2722 ts_tablet_manager.cc:1918] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:29:35.405555 2722 log.cc:1196] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/8939969c7276465c9aa342271065c15b
I20260620 18:29:35.405983 2722 ts_tablet_manager.cc:1939] T 8939969c7276465c9aa342271065c15b P f74c3f0649eb4012930065159a800b7a: Deleting consensus metadata
I20260620 18:29:35.406903 2122 catalog_manager.cc:4841] TS f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801): tablet 8939969c7276465c9aa342271065c15b (table test-table [id=651acc8a859543e79d33b343dd5cdeeb]) successfully deleted
I20260620 18:29:35.409664 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 (DEFAULT_TABLE table=test-table [id=11b8a7775e414b57b4b9f22f000aabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.409734 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 (DEFAULT_TABLE table=test-table [id=11b8a7775e414b57b4b9f22f000aabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.409880 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 (DEFAULT_TABLE table=test-table [id=11b8a7775e414b57b4b9f22f000aabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:35.410086 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381df53c08f474bb49a14ad3ec44b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.410296 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381df53c08f474bb49a14ad3ec44b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.410503 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381df53c08f474bb49a14ad3ec44b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:35.416978 2733 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Bootstrap starting.
I20260620 18:29:35.417331 2734 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Bootstrap starting.
I20260620 18:29:35.417441 2738 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Bootstrap starting.
I20260620 18:29:35.418996 2733 tablet_bootstrap.cc:654] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.419426 2738 tablet_bootstrap.cc:654] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.419476 2734 tablet_bootstrap.cc:654] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:35.429049 2733 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: No bootstrap required, opened a new log
I20260620 18:29:35.429225 2733 ts_tablet_manager.cc:1397] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20260620 18:29:35.429682 2738 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: No bootstrap required, opened a new log
I20260620 18:29:35.429908 2738 ts_tablet_manager.cc:1397] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 18:29:35.430068 2733 raft_consensus.cc:348] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.430325 2733 raft_consensus.cc:374] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.430418 2733 raft_consensus.cc:729] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c3f0649eb4012930065159a800b7a, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.430662 2733 consensus_queue.cc:260] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.431250 2733 ts_tablet_manager.cc:1428] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:29:35.431088 2738 raft_consensus.cc:348] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.431479 2738 raft_consensus.cc:374] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.431594 2738 raft_consensus.cc:729] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999831788cd8412696c34774394578c6, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.431859 2738 consensus_queue.cc:260] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.434573 2738 ts_tablet_manager.cc:1428] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:29:35.438149 2734 tablet_bootstrap.cc:492] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: No bootstrap required, opened a new log
I20260620 18:29:35.438335 2734 ts_tablet_manager.cc:1397] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260620 18:29:35.439085 2734 raft_consensus.cc:348] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.439349 2734 raft_consensus.cc:374] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:35.439442 2734 raft_consensus.cc:729] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f461ea159ba4d2eac934c3b417f68a5, State: Initialized, Role: FOLLOWER
I20260620 18:29:35.439713 2734 consensus_queue.cc:260] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.440287 2734 ts_tablet_manager.cc:1428] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:29:35.489395 2714 raft_consensus.cc:482] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:35.489610 2714 raft_consensus.cc:504] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.490314 2714 leader_election.cc:288] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801)
I20260620 18:29:35.490900 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381df53c08f474bb49a14ad3ec44b24" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5" is_pre_election: true
I20260620 18:29:35.490928 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381df53c08f474bb49a14ad3ec44b24" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a" is_pre_election: true
I20260620 18:29:35.491149 2437 raft_consensus.cc:2454] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 0.
I20260620 18:29:35.491149 2297 raft_consensus.cc:2454] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 0.
I20260620 18:29:35.491600 2504 leader_election.cc:302] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 999831788cd8412696c34774394578c6, f74c3f0649eb4012930065159a800b7a; no voters:
I20260620 18:29:35.491953 2714 raft_consensus.cc:2781] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:35.492076 2714 raft_consensus.cc:482] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:35.492179 2714 raft_consensus.cc:3037] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.495148 2714 raft_consensus.cc:504] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.495862 2714 leader_election.cc:288] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 election: Requested vote from peers 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929), f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801)
I20260620 18:29:35.496249 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381df53c08f474bb49a14ad3ec44b24" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f461ea159ba4d2eac934c3b417f68a5"
I20260620 18:29:35.496448 2297 raft_consensus.cc:3037] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.496533 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381df53c08f474bb49a14ad3ec44b24" candidate_uuid: "999831788cd8412696c34774394578c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c3f0649eb4012930065159a800b7a"
I20260620 18:29:35.496738 2437 raft_consensus.cc:3037] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:35.499544 2297 raft_consensus.cc:2454] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 1.
I20260620 18:29:35.499787 2437 raft_consensus.cc:2454] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999831788cd8412696c34774394578c6 in term 1.
I20260620 18:29:35.499977 2502 leader_election.cc:302] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f461ea159ba4d2eac934c3b417f68a5, 999831788cd8412696c34774394578c6; no voters:
I20260620 18:29:35.500509 2714 raft_consensus.cc:2781] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:35.500964 2714 raft_consensus.cc:686] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 1 LEADER]: Becoming Leader. State: Replica: 999831788cd8412696c34774394578c6, State: Running, Role: LEADER
I20260620 18:29:35.501237 2714 consensus_queue.cc:237] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } }
I20260620 18:29:35.505245 2134 catalog_manager.cc:5495] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 999831788cd8412696c34774394578c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "999831788cd8412696c34774394578c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999831788cd8412696c34774394578c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42137 } health_report { overall_health: HEALTHY } } }
I20260620 18:29:35.540293 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:35.535277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.0.97.3:42137)
0620 18:29:35.535430 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:35.535434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:35.535448 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:29:35.535493 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:35.535496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:35.535617 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:35.535713 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:35.536088 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.537279 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.538608 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:35.539008 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:35.539139 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:35.539580 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:35.539691 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:35.539793 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:35.539937 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 18:29:35.540008 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:29:35.550107 2437 raft_consensus.cc:1261] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Refusing update from remote peer 999831788cd8412696c34774394578c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:35.550897 2715 consensus_queue.cc:1035] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74c3f0649eb4012930065159a800b7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:35.551474 2298 raft_consensus.cc:1261] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Refusing update from remote peer 999831788cd8412696c34774394578c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:35.552248 2715 consensus_queue.cc:1035] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f461ea159ba4d2eac934c3b417f68a5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:29:35.577868 2704 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:29:35.581769 2705 mvcc.cc:205] Tried to move back new op lower bound from 7298990799047778304 to 7298990798857687040. Current Snapshot: MvccSnapshot[applied={T|T < 7298990799047778304}]
I20260620 18:29:37.163444 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:37.163771 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:37.168327 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 335906ac91c44e4dbd994f052fdfb416: Sending DeleteTablet for 3 replicas of tablet 8381df53c08f474bb49a14ad3ec44b24
I20260620 18:29:37.169162 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260620 18:29:37.169428 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56510
I20260620 18:29:37.169543 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8381df53c08f474bb49a14ad3ec44b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57664
I20260620 18:29:37.169972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2207
I20260620 18:29:37.181800 2787 tablet_replica.cc:330] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: stopping tablet replica
I20260620 18:29:37.182062 2787 raft_consensus.cc:2229] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:37.182319 2787 raft_consensus.cc:2258] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:37.182833 2788 tablet_replica.cc:330] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: stopping tablet replica
I20260620 18:29:37.183084 2788 raft_consensus.cc:2229] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:37.184180 2787 ts_tablet_manager.cc:1905] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:37.184285 2786 tablet_replica.cc:330] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: stopping tablet replica
I20260620 18:29:37.184489 2786 raft_consensus.cc:2229] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:37.184738 2786 raft_consensus.cc:2258] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:37.184753 2788 raft_consensus.cc:2258] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:37.185721 2788 ts_tablet_manager.cc:1905] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:37.185921 2786 ts_tablet_manager.cc:1905] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:37.191111 2787 ts_tablet_manager.cc:1918] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:37.191260 2787 log.cc:1196] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/8381df53c08f474bb49a14ad3ec44b24
I20260620 18:29:37.191745 2787 ts_tablet_manager.cc:1939] T 8381df53c08f474bb49a14ad3ec44b24 P f74c3f0649eb4012930065159a800b7a: Deleting consensus metadata
I20260620 18:29:37.192654 2788 ts_tablet_manager.cc:1918] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:37.192678 2786 ts_tablet_manager.cc:1918] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:29:37.192811 2786 log.cc:1196] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/8381df53c08f474bb49a14ad3ec44b24
I20260620 18:29:37.192817 2788 log.cc:1196] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/8381df53c08f474bb49a14ad3ec44b24
I20260620 18:29:37.192929 2122 catalog_manager.cc:4841] TS f74c3f0649eb4012930065159a800b7a (127.0.97.2:33801): tablet 8381df53c08f474bb49a14ad3ec44b24 (table test-table [id=11b8a7775e414b57b4b9f22f000aabd9]) successfully deleted
I20260620 18:29:37.193225 2786 ts_tablet_manager.cc:1939] T 8381df53c08f474bb49a14ad3ec44b24 P 3f461ea159ba4d2eac934c3b417f68a5: Deleting consensus metadata
I20260620 18:29:37.193256 2788 ts_tablet_manager.cc:1939] T 8381df53c08f474bb49a14ad3ec44b24 P 999831788cd8412696c34774394578c6: Deleting consensus metadata
I20260620 18:29:37.194118 2122 catalog_manager.cc:4841] TS 999831788cd8412696c34774394578c6 (127.0.97.3:42137): tablet 8381df53c08f474bb49a14ad3ec44b24 (table test-table [id=11b8a7775e414b57b4b9f22f000aabd9]) successfully deleted
I20260620 18:29:37.194144 2120 catalog_manager.cc:4841] TS 3f461ea159ba4d2eac934c3b417f68a5 (127.0.97.1:37929): tablet 8381df53c08f474bb49a14ad3ec44b24 (table test-table [id=11b8a7775e414b57b4b9f22f000aabd9]) successfully deleted
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:37.204711 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:37.172182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.97.1:37929)
0620 18:29:37.183002 (+ 10820us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:37.183009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:37.183038 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:29:37.183105 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:37.183112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:37.183198 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:37.183314 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:37.185795 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.186876 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.188773 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.189442 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.189556 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:37.198450 (+ 8894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:37.198474 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:37.198491 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:37.198534 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:37.201855 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.202545 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.202555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.202565 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.202697 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.203086 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.203090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.203092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.203430 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:37.203809 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:37.204110 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:29:37.204338 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10674,"thread_start_us":11345,"threads_started":1}
W20260620 18:29:37.205366 2338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59192
W20260620 18:29:37.205976 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:37.206162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2207
I20260620 18:29:37.227954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2347
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:37.243682 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:37.228734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.97.2:33801)
0620 18:29:37.229267 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:37.229272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:37.229288 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:29:37.229349 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:37.229352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:37.229413 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:37.229508 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:37.231106 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.231997 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.233225 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.233656 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.233760 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:37.238015 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:37.238029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:37.238044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:37.238080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:37.241013 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.241724 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.241738 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.241740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.241870 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.242235 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.242238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.242240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.242610 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:37.242850 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:37.243132 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:29:37.243290 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
W20260620 18:29:37.244328 2477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35116
W20260620 18:29:37.244971 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:37.245216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2347
I20260620 18:29:37.271250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2486
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:37.286283 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:37.272063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.97.3:42137)
0620 18:29:37.272624 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:37.272628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:37.272643 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:29:37.272693 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:37.272696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:37.272756 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:37.272873 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:37.274141 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.274861 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.276231 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:37.276666 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:37.276803 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:37.280974 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:37.280989 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:37.281005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:37.281035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:37.283653 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.284281 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.284297 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.284299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.284415 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:37.284790 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:37.284795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:37.284799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:37.285143 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:37.285434 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:37.285725 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:29:37.285939 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":242,"threads_started":1}
W20260620 18:29:37.287016 2616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45454
W20260620 18:29:37.287717 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2486: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:37.287927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2486
I20260620 18:29:37.309659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2104
I20260620 18:29:37.310822 2176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:37.425343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2104
2026-06-20T18:29:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8330 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2796](info): set up 2 sockets
Jun 20 18:29:37 dist-test-slave-mcnm krb5kdc[2796](info): commencing operation
krb5kdc: starting...
W20260620 18:29:38.508625 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:38 dist-test-slave-mcnm krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:38.712460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:39.040271 2809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:39.040591 2809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:39.040680 2809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:39.047873 2809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:39.047982 2809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:39.048027 2809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:39.048084 2809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:39.048127 2809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:39.058043 2809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:39.059321 2809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:39.060962 2809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:39.074510 2814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:39.074481 2817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:39.074412 2815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:39.075021 2809 server_base.cc:1034] running on GCE node
I20260620 18:29:39.076133 2809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:39.077728 2809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:39.079216 2809 hybrid_clock.cc:648] HybridClock initialized: now 1781980179079178 us; error 61 us; skew 500 ppm
I20260620 18:29:39.079667 2809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:39 dist-test-slave-mcnm krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:39.082998 2809 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:39.084599 2809 webserver.cc:460] Webserver started at http://127.0.97.62:33291/ using document root <none> and password file <none>
I20260620 18:29:39.085320 2809 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:39.085471 2809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:39.085811 2809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:39.088862 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "6426fe3180ff473d82ceaf42c14869a7"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "403e77f7bbadafe2ecf6e277e19a1fc1"
server_key_iv: "8ca3066950ce9c1e696b6e8a76655cb3"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.089663 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "6426fe3180ff473d82ceaf42c14869a7"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "403e77f7bbadafe2ecf6e277e19a1fc1"
server_key_iv: "8ca3066950ce9c1e696b6e8a76655cb3"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.095283 2809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:29:39.099181 2824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:39.100795 2809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:39.101022 2809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "6426fe3180ff473d82ceaf42c14869a7"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "403e77f7bbadafe2ecf6e277e19a1fc1"
server_key_iv: "8ca3066950ce9c1e696b6e8a76655cb3"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.101325 2809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:39.144810 2809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:39.149314 2809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:39.149578 2809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:39.168732 2809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33733
I20260620 18:29:39.168861 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33733 every 8 connection(s)
I20260620 18:29:39.170149 2809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:39.172183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2809
I20260620 18:29:39.172433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:39.173131 388 external_mini_cluster.cc:1428] Setting key 6a145ddd918785c8c6dcc85dcbb035eb
I20260620 18:29:39.174556 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:29:39 dist-test-slave-mcnm krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:39.189975 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7: Bootstrap starting.
I20260620 18:29:39.193606 2876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:39.194962 2876 log.cc:824] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:39.197479 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:39.175193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.97.62:33733)
0620 18:29:39.176026 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:39.176039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:39.176079 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:29:39.177656 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:39.177678 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:39.178099 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:39.178449 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:39.180373 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.181802 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.183651 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.184376 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.184518 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:39.189756 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:39.189764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:39.189780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:39.189819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:39.193005 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:39.193985 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:39.193991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:39.194001 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:39.194189 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:39.194602 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:39.194605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:39.194607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:39.195205 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:39.195430 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:39.196090 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 18:29:39.196532 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":393,"threads_started":1}
I20260620 18:29:39.198194 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7: No bootstrap required, opened a new log
I20260620 18:29:39.205616 2876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } }
I20260620 18:29:39.206115 2876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:39.206256 2876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6426fe3180ff473d82ceaf42c14869a7, State: Initialized, Role: FOLLOWER
I20260620 18:29:39.206918 2876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } }
I20260620 18:29:39.207132 2876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:39.207226 2876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:39.207352 2876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:39.209389 2876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } }
I20260620 18:29:39.209837 2876 leader_election.cc:302] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6426fe3180ff473d82ceaf42c14869a7; no voters:
I20260620 18:29:39.210417 2876 leader_election.cc:288] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:39.210565 2881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:39.211243 2881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6426fe3180ff473d82ceaf42c14869a7, State: Running, Role: LEADER
I20260620 18:29:39.211794 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } }
I20260620 18:29:39.212672 2876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:39.214721 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6426fe3180ff473d82ceaf42c14869a7. Latest consensus state: current_term: 1 leader_uuid: "6426fe3180ff473d82ceaf42c14869a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } } }
I20260620 18:29:39.214901 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6426fe3180ff473d82ceaf42c14869a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6426fe3180ff473d82ceaf42c14869a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33733 } } }
I20260620 18:29:39.215134 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:39.215184 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:39.216063 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:39.221807 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:39.230870 2888 catalog_manager.cc:1347] Generated new cluster ID: e771536c69794020a97835b5239bdb1e
I20260620 18:29:39.231000 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:39.251252 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:39.252694 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:39.270589 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6426fe3180ff473d82ceaf42c14869a7: Generated new TSK 0
I20260620 18:29:39.271462 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:39.332582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:39.685753 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:39.686043 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:39.686134 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:39.693360 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:39.693480 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:39.693598 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:39.704101 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:39.705404 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:39.706867 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:39.720098 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:39.720134 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:39.720214 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:39.721994 2902 server_base.cc:1034] running on GCE node
I20260620 18:29:39.722602 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:39.723624 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:39.724885 2902 hybrid_clock.cc:648] HybridClock initialized: now 1781980179724839 us; error 45 us; skew 500 ppm
I20260620 18:29:39.725240 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:39 dist-test-slave-mcnm krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:39.728785 2902 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:39.730424 2902 webserver.cc:460] Webserver started at http://127.0.97.1:41853/ using document root <none> and password file <none>
I20260620 18:29:39.731135 2902 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:39.731283 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:39.731663 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:39.734601 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "2704b04c626440b9b80244912f4dfd57"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "b4f79c37d2842ad603b83ecfbd050f34"
server_key_iv: "626b5533494dcdddac7dd10eb2b48501"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.735414 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2704b04c626440b9b80244912f4dfd57"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "b4f79c37d2842ad603b83ecfbd050f34"
server_key_iv: "626b5533494dcdddac7dd10eb2b48501"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.741192 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:39.744952 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:39.746417 2902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:39.746615 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "2704b04c626440b9b80244912f4dfd57"
format_stamp: "Formatted at 2026-06-20 18:29:39 on dist-test-slave-mcnm"
server_key: "b4f79c37d2842ad603b83ecfbd050f34"
server_key_iv: "626b5533494dcdddac7dd10eb2b48501"
server_key_version: "encryptionkey@0"
I20260620 18:29:39.746840 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:39.788865 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:39.793314 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:39.793622 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:39.794615 2902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:39.796103 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:39.796216 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:39.796367 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:39.796439 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:39.830190 2902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34549
I20260620 18:29:39.830248 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34549 every 8 connection(s)
I20260620 18:29:39.831501 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:39.834302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 2902
I20260620 18:29:39.834513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:39.835314 388 external_mini_cluster.cc:1428] Setting key 9eddb61df8ae00fc299214e5972f251e
Jun 20 18:29:39 dist-test-slave-mcnm krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:29:39.852118 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:39.834135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44471 (local address 127.0.97.62:33733)
0620 18:29:39.834669 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:39.834674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:39.835872 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 18:29:39.838157 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:39.838163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:39.838242 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:39.838366 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:39.839903 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.840597 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.842123 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.842524 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.842650 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:39.846073 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:39.846079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:39.846082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:39.846113 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:39.848769 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:39.849537 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:39.849542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:39.849544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:39.849662 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:39.850182 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:39.850186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:39.850188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:39.850546 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:39.850704 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:39.851485 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 18:29:39.851729 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260620 18:29:39.853267 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:39.834798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33733 (local address 127.0.97.1:44471)
0620 18:29:39.835699 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:39.835752 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:39.837897 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:39.838597 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:39.838605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:39.839099 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:39.839742 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:39.839761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.840753 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.840763 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:39.841982 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:39.841994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:39.842753 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:39.842761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:39.842998 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:39.843744 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:39.843765 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:39.845857 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:39.848928 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:39.848937 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:39.848956 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:39.849397 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:39.849791 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:39.849796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:39.849800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:39.850060 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:39.850683 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:39.850691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:39.851078 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 18:29:39.851380 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 18:29:39.852011 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:29:39.854988 3030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33733
I20260620 18:29:39.855386 3030 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:39.856321 3030 heartbeater.cc:511] Master 127.0.97.62:33733 requested a full tablet report, sending...
I20260620 18:29:39.858821 2841 ts_manager.cc:194] Registered new tserver with Master: 2704b04c626440b9b80244912f4dfd57 (127.0.97.1:34549)
I20260620 18:29:39.860765 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:39.879993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:40.206427 3036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:40.206756 3036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:40.206936 3036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:40.214305 3036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:40.214442 3036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:40.214628 3036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:40.224643 3036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:40.226017 3036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:40.227326 3036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:40.240370 3041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:40.240578 3044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:40.240597 3042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:40.241590 3036 server_base.cc:1034] running on GCE node
I20260620 18:29:40.242192 3036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:40.243126 3036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:40.244391 3036 hybrid_clock.cc:648] HybridClock initialized: now 1781980180244339 us; error 60 us; skew 500 ppm
I20260620 18:29:40.244726 3036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:40 dist-test-slave-mcnm krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:40.248003 3036 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:40.249449 3036 webserver.cc:460] Webserver started at http://127.0.97.2:40489/ using document root <none> and password file <none>
I20260620 18:29:40.250221 3036 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:40.250370 3036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:40.250694 3036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:40.253731 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "e1d3a4cacfc64fba85717899bd5d8a46"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "a9a5b4244f522606bc2850299f70e08b"
server_key_iv: "b1dfbf3220659ce34baa913d8c364326"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.254529 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e1d3a4cacfc64fba85717899bd5d8a46"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "a9a5b4244f522606bc2850299f70e08b"
server_key_iv: "b1dfbf3220659ce34baa913d8c364326"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.259902 3036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 18:29:40.263687 3051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.265141 3036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:40.265331 3036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "e1d3a4cacfc64fba85717899bd5d8a46"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "a9a5b4244f522606bc2850299f70e08b"
server_key_iv: "b1dfbf3220659ce34baa913d8c364326"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.265563 3036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:40.278156 3036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:40.282536 3036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:40.282840 3036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:40.283845 3036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:40.285135 3036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:40.285239 3036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.285406 3036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:40.285481 3036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.319386 3036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43873
I20260620 18:29:40.319473 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43873 every 8 connection(s)
I20260620 18:29:40.320669 3036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:40.328722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3036
I20260620 18:29:40.328919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:40.329391 388 external_mini_cluster.cc:1428] Setting key 838f9e0e65780c2c96027a03b55acaa1
Jun 20 18:29:40 dist-test-slave-mcnm krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:40.342439 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:40.323214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49679 (local address 127.0.97.62:33733)
0620 18:29:40.323397 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:40.323402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:40.324695 (+ 1293us) server_negotiation.cc:407] Connection header received
0620 18:29:40.326636 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:40.326642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:40.326741 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:40.326863 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:40.328508 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.329373 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.331463 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.331979 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.332079 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:40.336422 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:40.336427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:40.336431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:40.336464 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:40.339244 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:40.340137 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:40.340143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:40.340147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:40.340296 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:40.340732 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:40.340737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:40.340740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:40.341008 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:40.341160 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:40.341849 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:29:40.342067 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:29:40.343492 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:40.323728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33733 (local address 127.0.97.2:49679)
0620 18:29:40.324479 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:40.324550 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:40.326350 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:40.327098 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:40.327106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:40.327714 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:40.328284 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:40.328302 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.329534 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.329541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:40.331316 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:40.331334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.332217 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.332225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:40.332536 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:40.333361 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:40.333395 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:40.336171 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:40.339394 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:40.339404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:40.339440 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:40.340003 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:40.340414 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:40.340419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:40.340423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:40.340610 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:40.341135 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:40.341143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:40.341494 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 18:29:40.341763 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 18:29:40.342314 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
I20260620 18:29:40.345075 3164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33733
I20260620 18:29:40.345430 3164 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:40.346123 3164 heartbeater.cc:511] Master 127.0.97.62:33733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:29:40.347611 2841 ts_manager.cc:194] Registered new tserver with Master: e1d3a4cacfc64fba85717899bd5d8a46 (127.0.97.2:43873)
I20260620 18:29:40.348454 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:40.373078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:40.691026 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:40.691319 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:40.691470 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:40.699378 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:40.699492 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:40.699718 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:40.709604 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:40.710870 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:40.712237 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:40.725320 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:40.725461 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:40.725461 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:40.726172 3169 server_base.cc:1034] running on GCE node
I20260620 18:29:40.726732 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:40.727634 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:40.728840 3169 hybrid_clock.cc:648] HybridClock initialized: now 1781980180728775 us; error 56 us; skew 500 ppm
I20260620 18:29:40.729153 3169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:40 dist-test-slave-mcnm krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:40.732375 3169 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:40.733938 3169 webserver.cc:460] Webserver started at http://127.0.97.3:39801/ using document root <none> and password file <none>
I20260620 18:29:40.734634 3169 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:40.734731 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:40.735071 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:40.737934 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "ab62d6f8bbfe4f68b70ef7fa0d94123c"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "6f6d96898d7a7c58f54c58128c88703f"
server_key_iv: "4ffa36f5c92e03c47c96c1e7b5233e53"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.738695 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ab62d6f8bbfe4f68b70ef7fa0d94123c"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "6f6d96898d7a7c58f54c58128c88703f"
server_key_iv: "4ffa36f5c92e03c47c96c1e7b5233e53"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.743685 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:29:40.747404 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.748890 3169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:40.749022 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "ab62d6f8bbfe4f68b70ef7fa0d94123c"
format_stamp: "Formatted at 2026-06-20 18:29:40 on dist-test-slave-mcnm"
server_key: "6f6d96898d7a7c58f54c58128c88703f"
server_key_iv: "4ffa36f5c92e03c47c96c1e7b5233e53"
server_key_version: "encryptionkey@0"
I20260620 18:29:40.749176 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:40.770674 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:40.775465 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:40.775792 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:40.776757 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:40.778111 3169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:40.778183 3169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.778319 3169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:40.778368 3169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:40.810976 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42795
I20260620 18:29:40.811009 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42795 every 8 connection(s)
I20260620 18:29:40.812279 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:40.821357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3169
I20260620 18:29:40.821575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:40.822131 388 external_mini_cluster.cc:1428] Setting key 4547bca3a7505672df667238a6a25a15
Jun 20 18:29:40 dist-test-slave-mcnm krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:40.833688 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:40.815056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42255 (local address 127.0.97.62:33733)
0620 18:29:40.815225 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:40.815229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:40.816385 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 18:29:40.818184 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:40.818189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:40.818275 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:40.818378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:40.820057 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.820856 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.823322 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.823905 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.824048 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:40.827448 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:40.827452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:40.827456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:40.827487 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:40.830209 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:40.831181 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:40.831186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:40.831189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:40.831329 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:40.831903 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:40.831907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:40.831909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:40.832191 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:40.832308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:40.833182 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 18:29:40.833355 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:29:40.834923 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:40.815420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33733 (local address 127.0.97.3:42255)
0620 18:29:40.816172 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:40.816218 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:40.817893 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:40.818641 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:40.818649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:40.819171 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:40.819864 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:40.819884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.821031 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.821039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:40.822994 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:40.823006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:40.824154 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:40.824162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:40.824408 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:40.825110 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:40.825136 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:40.827245 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:40.830353 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:40.830362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:40.830378 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:40.831016 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:40.831453 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:40.831457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:40.831461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:40.831729 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:40.832328 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:40.832335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:40.832744 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 18:29:40.833053 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 18:29:40.833593 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":264,"threads_started":1}
I20260620 18:29:40.836591 3297 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33733
I20260620 18:29:40.836947 3297 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:40.837882 3297 heartbeater.cc:511] Master 127.0.97.62:33733 requested a full tablet report, sending...
I20260620 18:29:40.839334 2841 ts_manager.cc:194] Registered new tserver with Master: ab62d6f8bbfe4f68b70ef7fa0d94123c (127.0.97.3:42795)
I20260620 18:29:40.840198 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42255
I20260620 18:29:40.850674 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:40.863893 3030 heartbeater.cc:503] Master 127.0.97.62:33733 was elected leader, sending a full tablet report...
W20260620 18:29:40.883178 3032 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:29:40.880567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.0.97.62:33733)
0620 18:29:40.880791 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:40.880814 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:40.881322 (+ 508us) server_negotiation.cc:407] Connection header received
0620 18:29:40.881467 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:40.881471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:40.881538 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:40.881635 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:40.882987 (+ 1352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42590: BlockingRecv error: recv got EOF from 127.0.0.1:42590 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 18:29:40.886699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2902
W20260620 18:29:40.889762 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2902: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34549: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:29:40.889940 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2902
I20260620 18:29:40.905615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3036
W20260620 18:29:40.908511 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3036: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43873: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:29:40.908684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3036
I20260620 18:29:40.924608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3169
W20260620 18:29:40.927754 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3169: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42795: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:29:40.927927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3169
I20260620 18:29:40.944290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 2809
I20260620 18:29:40.945709 2871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:41.048125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 2809
2026-06-20T18:29:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3620 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:41 dist-test-slave-mcnm krb5kdc[3314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:41 dist-test-slave-mcnm krb5kdc[3314](info): set up 2 sockets
Jun 20 18:29:41 dist-test-slave-mcnm krb5kdc[3314](info): commencing operation
krb5kdc: starting...
W20260620 18:29:42.211918 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:42 dist-test-slave-mcnm krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:42.407200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42075
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:42.718508 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:42.718860 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:42.719033 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:42.726343 3327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:42.726504 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:42.726617 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:42.726727 3327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:42.726820 3327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:42.736858 3327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:42.738197 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:42.739542 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:42.751303 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:42.751375 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:42.751610 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:42.752336 3327 server_base.cc:1034] running on GCE node
I20260620 18:29:42.753036 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:42.754400 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:42.755664 3327 hybrid_clock.cc:648] HybridClock initialized: now 1781980182755584 us; error 69 us; skew 500 ppm
I20260620 18:29:42.756013 3327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:42 dist-test-slave-mcnm krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:42.766250 3327 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:42.768579 3327 webserver.cc:460] Webserver started at http://127.0.97.62:38577/ using document root <none> and password file <none>
I20260620 18:29:42.769567 3327 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:42.769737 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:42.770123 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:42.773550 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "2beadcb987084f2299abc309810527c9"
format_stamp: "Formatted at 2026-06-20 18:29:42 on dist-test-slave-mcnm"
server_key: "aa7f4631edbd1a682402faaa33719ec7"
server_key_iv: "ccb9c2c3f440d3eb5328667420e110ef"
server_key_version: "encryptionkey@0"
I20260620 18:29:42.774274 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "2beadcb987084f2299abc309810527c9"
format_stamp: "Formatted at 2026-06-20 18:29:42 on dist-test-slave-mcnm"
server_key: "aa7f4631edbd1a682402faaa33719ec7"
server_key_iv: "ccb9c2c3f440d3eb5328667420e110ef"
server_key_version: "encryptionkey@0"
I20260620 18:29:42.779458 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:29:42.783213 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:42.784807 3327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:42.785012 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "2beadcb987084f2299abc309810527c9"
format_stamp: "Formatted at 2026-06-20 18:29:42 on dist-test-slave-mcnm"
server_key: "aa7f4631edbd1a682402faaa33719ec7"
server_key_iv: "ccb9c2c3f440d3eb5328667420e110ef"
server_key_version: "encryptionkey@0"
I20260620 18:29:42.785254 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:42.811784 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:42.816286 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:42.816609 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:42.835379 3327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42075
I20260620 18:29:42.835408 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42075 every 8 connection(s)
I20260620 18:29:42.836869 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:42.840735 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:42.845826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3327
I20260620 18:29:42.846047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:42.846802 388 external_mini_cluster.cc:1428] Setting key 80556c1bc79730420e28d080195bb4ed
I20260620 18:29:42.852242 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9: Bootstrap starting.
I20260620 18:29:42.855979 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:42.857227 3394 log.cc:824] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:42.861148 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9: No bootstrap required, opened a new log
Jun 20 18:29:42 dist-test-slave-mcnm krb5kdc[3314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:42.869568 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } }
I20260620 18:29:42.869973 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:42.870146 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2beadcb987084f2299abc309810527c9, State: Initialized, Role: FOLLOWER
I20260620 18:29:42.870735 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } }
I20260620 18:29:42.870973 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:42.871124 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:42.871302 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:42.871479 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:42.848802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.97.62:42075)
0620 18:29:42.849679 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:42.849694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:42.849739 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:29:42.851440 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:42.851464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:42.851942 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:42.852288 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:42.854085 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:42.855372 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:42.857189 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:42.858248 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:42.858472 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:42.863046 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:42.863054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:42.863072 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:42.863114 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:42.866931 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:42.867694 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:42.867701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:42.867708 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:42.867893 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:42.868300 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:42.868304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:42.868306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:42.868875 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:42.869188 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:42.869656 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 18:29:42.870169 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":426,"threads_started":1}
I20260620 18:29:42.874207 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } }
I20260620 18:29:42.874782 3394 leader_election.cc:302] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2beadcb987084f2299abc309810527c9; no voters:
I20260620 18:29:42.875442 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:29:42.875445 3394 leader_election.cc:288] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:42.875571 3399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:42.877470 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:42.880086 3399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2beadcb987084f2299abc309810527c9, State: Running, Role: LEADER
I20260620 18:29:42.880615 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } }
I20260620 18:29:42.883085 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2beadcb987084f2299abc309810527c9. Latest consensus state: current_term: 1 leader_uuid: "2beadcb987084f2299abc309810527c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } } }
I20260620 18:29:42.883427 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:42.883946 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2beadcb987084f2299abc309810527c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beadcb987084f2299abc309810527c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42075 } } }
I20260620 18:29:42.884282 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:42.887670 3406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:42.893589 3406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:42.916275 3406 catalog_manager.cc:1347] Generated new cluster ID: 97acbbedbc5f456994a3331cfebf4937
I20260620 18:29:42.916409 3406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:42.944751 3406 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:42.948074 3406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:42.957238 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:42.941788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.97.62:42075)
0620 18:29:42.941940 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:42.941946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:42.948450 (+ 6504us) server_negotiation.cc:407] Connection header received
0620 18:29:42.948942 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:42.948962 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:42.949054 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:42.949151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:42.950480 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:42.951595 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:42.953403 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:42.953987 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:42.954073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:42.955447 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:42.955454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:42.955459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:42.956371 (+ 912us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:42.956449 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:42.956486 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:42.956564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:42.956803 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 18:29:42.956898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:29:42.959303 3406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2beadcb987084f2299abc309810527c9: Generated new TSK 0
I20260620 18:29:42.960357 3406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:29:42.974694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3327
I20260620 18:29:42.976372 3389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:43.066102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3327
2026-06-20T18:29:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2011 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:43 dist-test-slave-mcnm krb5kdc[3429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:43 dist-test-slave-mcnm krb5kdc[3429](info): set up 2 sockets
Jun 20 18:29:43 dist-test-slave-mcnm krb5kdc[3429](info): commencing operation
krb5kdc: starting...
W20260620 18:29:44.143961 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:44 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:44.339803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:44.668288 3442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:44.668624 3442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:44.668825 3442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:44.676896 3442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:44.677006 3442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:44.677104 3442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:44.677183 3442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:44.677248 3442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:44.687530 3442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:44.688956 3442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:44.690346 3442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:44.702702 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:44.702791 3447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:44.702764 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:44.704007 3442 server_base.cc:1034] running on GCE node
I20260620 18:29:44.705118 3442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:44.706717 3442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:44.707981 3442 hybrid_clock.cc:648] HybridClock initialized: now 1781980184707907 us; error 52 us; skew 500 ppm
I20260620 18:29:44.708374 3442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:44 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:44.711772 3442 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:44.713302 3442 webserver.cc:460] Webserver started at http://127.0.97.62:42275/ using document root <none> and password file <none>
I20260620 18:29:44.714010 3442 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:44.714138 3442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:44.714463 3442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:44.717533 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3"
format_stamp: "Formatted at 2026-06-20 18:29:44 on dist-test-slave-mcnm"
server_key: "7867ed3bd2320ff8a69887f6bb7f8f44"
server_key_iv: "4785f5b11dda2e41be3fec5bd00d2855"
server_key_version: "encryptionkey@0"
I20260620 18:29:44.718338 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3"
format_stamp: "Formatted at 2026-06-20 18:29:44 on dist-test-slave-mcnm"
server_key: "7867ed3bd2320ff8a69887f6bb7f8f44"
server_key_iv: "4785f5b11dda2e41be3fec5bd00d2855"
server_key_version: "encryptionkey@0"
I20260620 18:29:44.725350 3442 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 18:29:44.729121 3457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:44.730655 3442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:29:44.730832 3442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3"
format_stamp: "Formatted at 2026-06-20 18:29:44 on dist-test-slave-mcnm"
server_key: "7867ed3bd2320ff8a69887f6bb7f8f44"
server_key_iv: "4785f5b11dda2e41be3fec5bd00d2855"
server_key_version: "encryptionkey@0"
I20260620 18:29:44.731045 3442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:44.783752 3442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:44.788355 3442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:44.788676 3442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:44.809055 3442 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35229
I20260620 18:29:44.809162 3508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35229 every 8 connection(s)
I20260620 18:29:44.810547 3442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:44.814400 3509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:44.819736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3442
I20260620 18:29:44.819937 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:44.820710 388 external_mini_cluster.cc:1428] Setting key 524dc711f81825d28cb2addc9155a56e
I20260620 18:29:44.825863 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3: Bootstrap starting.
I20260620 18:29:44.829478 3509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:44.830700 3509 log.cc:824] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:44.834259 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3: No bootstrap required, opened a new log
Jun 20 18:29:44 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:44.842147 3509 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260620 18:29:44.842540 3509 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:44.842679 3509 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bd5940eefb4a5d8c44a65d7c4848c3, State: Initialized, Role: FOLLOWER
I20260620 18:29:44.843389 3509 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260620 18:29:44.843679 3509 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:44.843865 3509 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:44.844053 3509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:44.845095 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:44.822836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.97.62:35229)
0620 18:29:44.823734 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:44.823749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:44.823792 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:29:44.825394 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:44.825416 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:44.825803 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:44.826184 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:44.827850 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:44.829140 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:44.831000 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:44.832142 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:44.832301 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:44.837216 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:44.837225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:44.837243 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:44.837344 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:44.840612 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:44.841361 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:44.841368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:44.841378 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:44.841547 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:44.841942 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:44.841948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:44.841951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:44.842465 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:44.843033 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:44.843348 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:29:44.843861 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":443,"threads_started":1}
I20260620 18:29:44.846482 3509 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260620 18:29:44.846982 3509 leader_election.cc:302] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2bd5940eefb4a5d8c44a65d7c4848c3; no voters:
I20260620 18:29:44.847635 3509 leader_election.cc:288] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:44.847764 3514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:44.849025 3514 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [term 1 LEADER]: Becoming Leader. State: Replica: d2bd5940eefb4a5d8c44a65d7c4848c3, State: Running, Role: LEADER
I20260620 18:29:44.849572 3514 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260620 18:29:44.849790 3509 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:44.852636 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2bd5940eefb4a5d8c44a65d7c4848c3. Latest consensus state: current_term: 1 leader_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } } }
I20260620 18:29:44.853050 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:44.853607 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bd5940eefb4a5d8c44a65d7c4848c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } } }
I20260620 18:29:44.853993 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:44.854852 3521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:44.860785 3521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:44.870127 3521 catalog_manager.cc:1347] Generated new cluster ID: 9cafea576b3f431487ca5855000cb8a9
I20260620 18:29:44.870313 3521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:44.906554 3521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:44.908026 3521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:44.917858 3521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2bd5940eefb4a5d8c44a65d7c4848c3: Generated new TSK 0
I20260620 18:29:44.918754 3521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:44.975685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:45.327958 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:45.328241 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:45.328418 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:45.335793 3535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:45.335983 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:45.336182 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:45.346127 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:45.347704 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:45.349038 3535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:45.362972 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:45.363401 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:45.363440 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:45.364284 3535 server_base.cc:1034] running on GCE node
I20260620 18:29:45.365027 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:45.366097 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:45.367347 3535 hybrid_clock.cc:648] HybridClock initialized: now 1781980185367300 us; error 56 us; skew 500 ppm
I20260620 18:29:45.367753 3535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:45 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:45.371163 3535 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:45.372864 3535 webserver.cc:460] Webserver started at http://127.0.97.1:38829/ using document root <none> and password file <none>
I20260620 18:29:45.373696 3535 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:45.373835 3535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:45.374169 3535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:45.377343 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "8f41b23a41704dc497e79bd4dcdb5d18"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "f3a5df637f94087e839cc5ce1fba9274"
server_key_iv: "5ea49cc40dbb3df527ea689b9a478440"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.378204 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f41b23a41704dc497e79bd4dcdb5d18"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "f3a5df637f94087e839cc5ce1fba9274"
server_key_iv: "5ea49cc40dbb3df527ea689b9a478440"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.383828 3535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 18:29:45.387676 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:45.389093 3535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:45.389326 3535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "8f41b23a41704dc497e79bd4dcdb5d18"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "f3a5df637f94087e839cc5ce1fba9274"
server_key_iv: "5ea49cc40dbb3df527ea689b9a478440"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.389554 3535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:45.431277 3535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:45.436019 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:45.436362 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:45.437325 3535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:45.438666 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:45.438740 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:45.438864 3535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:45.438931 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:45.472240 3535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40161
I20260620 18:29:45.472286 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40161 every 8 connection(s)
I20260620 18:29:45.473522 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:45.475797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3535
I20260620 18:29:45.476009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:45.476831 388 external_mini_cluster.cc:1428] Setting key d98ff54955be2254a9b6efe43590b85e
Jun 20 18:29:45 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:29:45.498875 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:45.476199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34987 (local address 127.0.97.62:35229)
0620 18:29:45.476659 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:45.476665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:45.477738 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 18:29:45.480188 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:45.480193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:45.480285 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:45.480393 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:45.482604 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.483409 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.485524 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.486358 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.486517 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:45.490770 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:45.490781 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:45.490790 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:45.490843 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:45.495107 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:45.496069 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:45.496078 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:45.496083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:45.496255 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:45.496752 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:45.496762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:45.496766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:45.497176 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:45.497360 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:45.498162 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 18:29:45.498437 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260620 18:29:45.499876 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:45.476774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.1:34987)
0620 18:29:45.477575 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:45.477619 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:45.479890 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:45.480698 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:45.480713 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:45.481522 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:45.482405 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:45.482433 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.483613 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.483621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:45.485344 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:45.485356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.486637 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.486642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:45.486845 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:45.487738 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:45.487764 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:45.490492 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:45.495304 (+ 4812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:45.495312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:45.495326 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:45.495904 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:45.496389 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:45.496393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:45.496396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:45.496606 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:45.497319 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:45.497325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:45.497677 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:29:45.497978 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 18:29:45.498531 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:29:45.501822 3663 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260620 18:29:45.502166 3663 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:45.503104 3663 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260620 18:29:45.505721 3474 ts_manager.cc:194] Registered new tserver with Master: 8f41b23a41704dc497e79bd4dcdb5d18 (127.0.97.1:40161)
I20260620 18:29:45.507381 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:45.527465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:45.854790 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:45.855187 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:45.855367 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:45.862849 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:45.863018 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:45.863204 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:45.873432 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:45.874799 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:45.876202 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:45.889678 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:45.889978 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:45.889928 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:45.891261 3669 server_base.cc:1034] running on GCE node
I20260620 18:29:45.891949 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:45.892968 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:45.894263 3669 hybrid_clock.cc:648] HybridClock initialized: now 1781980185894137 us; error 82 us; skew 500 ppm
I20260620 18:29:45.894646 3669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:45 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:45.898132 3669 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:45.899910 3669 webserver.cc:460] Webserver started at http://127.0.97.2:45747/ using document root <none> and password file <none>
I20260620 18:29:45.900676 3669 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:45.900833 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:45.901156 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:45.904385 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "7f1580dcffd4429a882180182ec57cfb"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "7f46c5b310368789e9787516c0859aa1"
server_key_iv: "557b5370115ac072847a3d1f82f639f1"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.905371 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7f1580dcffd4429a882180182ec57cfb"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "7f46c5b310368789e9787516c0859aa1"
server_key_iv: "557b5370115ac072847a3d1f82f639f1"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.910961 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 18:29:45.914752 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:45.916405 3669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:45.916584 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "7f1580dcffd4429a882180182ec57cfb"
format_stamp: "Formatted at 2026-06-20 18:29:45 on dist-test-slave-mcnm"
server_key: "7f46c5b310368789e9787516c0859aa1"
server_key_iv: "557b5370115ac072847a3d1f82f639f1"
server_key_version: "encryptionkey@0"
I20260620 18:29:45.916785 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:45.943970 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:45.948594 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:45.948921 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:45.949963 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:45.951489 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:45.951612 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 18:29:45.951763 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:45.951833 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:45.986136 3669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42585
I20260620 18:29:45.986193 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42585 every 8 connection(s)
I20260620 18:29:45.987444 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:45.997036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3669
I20260620 18:29:45.997306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:45.997838 388 external_mini_cluster.cc:1428] Setting key 556cef993a1cada3c3525f3ceaafb08b
Jun 20 18:29:45 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:46.009168 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:45.990202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58961 (local address 127.0.97.62:35229)
0620 18:29:45.990513 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:45.990518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:45.991708 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 18:29:45.993666 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:45.993672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:45.993768 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:45.993885 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:45.995462 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.996220 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.997922 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.998405 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.998547 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.002443 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.002454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.002461 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.002510 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.005763 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.006610 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.006619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.006626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.006784 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.007182 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.007190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.007196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.007596 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.007752 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.008442 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.008696 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260620 18:29:46.009835 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:45.990726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.2:58961)
0620 18:29:45.991472 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:45.991582 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:45.993334 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:45.994140 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:45.994149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:45.994679 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:45.995280 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:45.995298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.996383 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.996396 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:45.997759 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:45.997775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:45.998705 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:45.998714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:45.999021 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:45.999840 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:45.999863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:46.002186 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:46.005914 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:46.005924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:46.005944 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:46.006474 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:46.006903 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:46.006906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:46.006909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:46.007076 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:46.007730 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:46.007738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:46.008070 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:29:46.008310 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 18:29:46.008695 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260620 18:29:46.011358 3797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260620 18:29:46.011703 3797 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:46.012411 3797 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260620 18:29:46.013906 3474 ts_manager.cc:194] Registered new tserver with Master: 7f1580dcffd4429a882180182ec57cfb (127.0.97.2:42585)
I20260620 18:29:46.014707 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58961
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:46.042485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:46.369037 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:46.369329 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:46.369515 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:46.376763 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:46.376874 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:46.377048 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:46.386652 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:46.388022 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:46.389365 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:46.402151 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:46.402194 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:46.402547 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:46.403494 3802 server_base.cc:1034] running on GCE node
I20260620 18:29:46.404089 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:46.405071 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:46.406292 3802 hybrid_clock.cc:648] HybridClock initialized: now 1781980186406249 us; error 46 us; skew 500 ppm
I20260620 18:29:46.406569 3802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:46.409817 3802 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:46.411196 3802 webserver.cc:460] Webserver started at http://127.0.97.3:45535/ using document root <none> and password file <none>
I20260620 18:29:46.411947 3802 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:46.412048 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:46.412339 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:46.415473 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "0ef456f3f6f444e7aa04771022111e39"
format_stamp: "Formatted at 2026-06-20 18:29:46 on dist-test-slave-mcnm"
server_key: "0f4d192e301cc7eb5f1c19cd51722162"
server_key_iv: "b5ce4c83b140165ca63c1bb6fba0e4f2"
server_key_version: "encryptionkey@0"
I20260620 18:29:46.416353 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0ef456f3f6f444e7aa04771022111e39"
format_stamp: "Formatted at 2026-06-20 18:29:46 on dist-test-slave-mcnm"
server_key: "0f4d192e301cc7eb5f1c19cd51722162"
server_key_iv: "b5ce4c83b140165ca63c1bb6fba0e4f2"
server_key_version: "encryptionkey@0"
I20260620 18:29:46.422448 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:29:46.425871 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:46.427222 3802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:29:46.427343 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "0ef456f3f6f444e7aa04771022111e39"
format_stamp: "Formatted at 2026-06-20 18:29:46 on dist-test-slave-mcnm"
server_key: "0f4d192e301cc7eb5f1c19cd51722162"
server_key_iv: "b5ce4c83b140165ca63c1bb6fba0e4f2"
server_key_version: "encryptionkey@0"
I20260620 18:29:46.427503 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:46.443357 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:46.447566 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:46.447801 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:46.448652 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:46.449854 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:46.449923 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:46.450018 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:46.450065 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:46.482244 3802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45067
I20260620 18:29:46.482327 3929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45067 every 8 connection(s)
I20260620 18:29:46.483455 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:46.491395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3802
I20260620 18:29:46.491621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:46.492156 388 external_mini_cluster.cc:1428] Setting key 256733041a36edc1753633e77b580b48
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:46.504043 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.486289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49891 (local address 127.0.97.62:35229)
0620 18:29:46.486514 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:46.486522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:46.487649 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 18:29:46.489434 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:46.489438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:46.489516 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:46.489608 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:46.491055 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.492346 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.494318 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.494923 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.495050 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.498541 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.498545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.498549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.498580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.501263 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.501973 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.501977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.501980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.502117 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.502543 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.502546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.502548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.502808 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.502917 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.503634 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.503795 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:29:46.504995 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.486717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.3:49891)
0620 18:29:46.487443 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:46.487480 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:46.489172 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:46.489834 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:46.489841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:46.490312 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:46.490861 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:46.490877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.492492 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.492496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:46.494100 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:46.494138 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.495162 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.495173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:46.495476 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.496170 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:46.496196 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:46.498336 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:46.501390 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:46.501397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:46.501410 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:46.501848 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:46.502253 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:46.502256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:46.502259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:46.502443 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:46.502944 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:46.502949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:46.503248 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 18:29:46.503472 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 18:29:46.503904 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260620 18:29:46.506667 3930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260620 18:29:46.506970 3930 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:46.507753 3930 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260620 18:29:46.509104 3474 ts_manager.cc:194] Registered new tserver with Master: 0ef456f3f6f444e7aa04771022111e39 (127.0.97.3:45067)
I20260620 18:29:46.509861 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49891
I20260620 18:29:46.510453 3663 heartbeater.cc:503] Master 127.0.97.62:35229 was elected leader, sending a full tablet report...
I20260620 18:29:46.519753 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:46.567411 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.553079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.97.62:35229)
0620 18:29:46.553397 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:46.553407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:46.553698 (+ 291us) server_negotiation.cc:407] Connection header received
0620 18:29:46.553939 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:46.553943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:46.554006 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:46.554114 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:46.555473 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.556333 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.557582 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.557998 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.558134 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.562244 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.562250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.562253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.562289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.564832 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.565472 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.565484 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.565487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.565598 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.565971 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.565977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.565981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.566301 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.566544 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.566842 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.567093 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
W20260620 18:29:46.568030 3474 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32926
I20260620 18:29:46.569640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3535
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:46.585250 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.570682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.0.97.1:40161)
0620 18:29:46.571312 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:46.571319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:46.571343 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:29:46.571396 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:46.571402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:46.571703 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:46.571835 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:46.572954 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.574104 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.575272 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.575912 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.576015 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.579729 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.579740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.579755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.579791 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.582692 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.583377 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.583383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.583385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.583503 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.583867 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.583870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.583872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.584183 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.584408 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.584613 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.584817 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
W20260620 18:29:46.586472 3658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51548
W20260620 18:29:46.587568 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:46.587770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3535
I20260620 18:29:46.604056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3669
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:46.619593 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.604993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.0.97.2:42585)
0620 18:29:46.605515 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:46.605522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:46.605545 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:29:46.605601 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:46.605608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:46.605832 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:46.605994 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:46.607177 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.608286 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.609510 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.610084 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.610230 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.614382 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.614393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.614407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.614440 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.617149 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.617747 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.617753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.617755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.617864 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.618288 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.618291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.618293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.618595 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.618813 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.619075 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.619260 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
W20260620 18:29:46.620935 3792 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43340
W20260620 18:29:46.622017 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3669: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:46.622138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3669
I20260620 18:29:46.637843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3802
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:46.655635 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:46.638926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.0.97.3:45067)
0620 18:29:46.639672 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:46.639680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:46.639709 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:29:46.639770 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:46.639777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:46.639995 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:46.640182 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:46.641623 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.642726 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.644186 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:46.644808 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:46.644960 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:46.650012 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:46.650024 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:46.650039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:46.650074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:46.652901 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.653578 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.653584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.653587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.653712 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:46.654075 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:46.654078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:46.654080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:46.654399 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:46.654619 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:46.655070 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:29:46.655279 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":343,"threads_started":1}
W20260620 18:29:46.657027 3925 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50602
W20260620 18:29:46.658146 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:29:46.658318 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3802
I20260620 18:29:46.674788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3442
I20260620 18:29:46.676045 3504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:46.781308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3442
2026-06-20T18:29:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3720 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3947](info): set up 2 sockets
Jun 20 18:29:46 dist-test-slave-mcnm krb5kdc[3947](info): commencing operation
krb5kdc: starting...
W20260620 18:29:47.864279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:47 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:48.055335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38515
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:48.369002 3960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:48.369266 3960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:48.369446 3960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:48.376339 3960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:48.376437 3960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:48.376526 3960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:48.376636 3960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:48.376703 3960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:48.386042 3960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:48.387284 3960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:48.388585 3960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:48.400650 3965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:48.400616 3966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:48.400678 3968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:48.401307 3960 server_base.cc:1034] running on GCE node
I20260620 18:29:48.402051 3960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:48.403334 3960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:48.404538 3960 hybrid_clock.cc:648] HybridClock initialized: now 1781980188404474 us; error 63 us; skew 500 ppm
I20260620 18:29:48.404927 3960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:48 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:48.411219 3960 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:48.412773 3960 webserver.cc:460] Webserver started at http://127.0.97.62:38029/ using document root <none> and password file <none>
I20260620 18:29:48.413446 3960 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:48.413547 3960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:48.413803 3960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:48.416757 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "584f9fedda9844caa57536d0c301dae6"
format_stamp: "Formatted at 2026-06-20 18:29:48 on dist-test-slave-mcnm"
server_key: "5957eaef48c22dc5cb6e01a3b19ee8ba"
server_key_iv: "d56f5d88bf4b6c83abe21b607aff625e"
server_key_version: "encryptionkey@0"
I20260620 18:29:48.417482 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "584f9fedda9844caa57536d0c301dae6"
format_stamp: "Formatted at 2026-06-20 18:29:48 on dist-test-slave-mcnm"
server_key: "5957eaef48c22dc5cb6e01a3b19ee8ba"
server_key_iv: "d56f5d88bf4b6c83abe21b607aff625e"
server_key_version: "encryptionkey@0"
I20260620 18:29:48.422559 3960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:29:48.426199 3975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:48.427734 3960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:48.427917 3960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "584f9fedda9844caa57536d0c301dae6"
format_stamp: "Formatted at 2026-06-20 18:29:48 on dist-test-slave-mcnm"
server_key: "5957eaef48c22dc5cb6e01a3b19ee8ba"
server_key_iv: "d56f5d88bf4b6c83abe21b607aff625e"
server_key_version: "encryptionkey@0"
I20260620 18:29:48.428122 3960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:48.450423 3960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:48.454517 3960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:48.454818 3960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:48.473646 3960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38515
I20260620 18:29:48.473670 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38515 every 8 connection(s)
I20260620 18:29:48.474979 3960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:48.478737 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:48.482450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 3960
I20260620 18:29:48.482692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:48.483336 388 external_mini_cluster.cc:1428] Setting key 737dc0c562e807efe1442b899bb4c290
I20260620 18:29:48.489161 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6: Bootstrap starting.
I20260620 18:29:48.493198 4027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:48.494501 4027 log.cc:824] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:48.498078 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6: No bootstrap required, opened a new log
Jun 20 18:29:48 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:48.507112 4027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } }
I20260620 18:29:48.507511 4027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:48.507695 4027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584f9fedda9844caa57536d0c301dae6, State: Initialized, Role: FOLLOWER
I20260620 18:29:48.508176 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:48.485444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.0.97.62:38515)
0620 18:29:48.486343 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:48.486355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:48.486397 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:29:48.488289 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:48.488311 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:48.488703 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:48.489055 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:48.490989 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:48.492393 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:48.494351 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:48.495252 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:48.495454 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:48.500322 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:48.500329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:48.500349 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:48.500387 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:48.504000 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:48.504697 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:48.504705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:48.504719 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:48.504890 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:48.505265 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:48.505270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:48.505274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:48.505787 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:48.506112 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:48.506516 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 18:29:48.506942 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":448,"threads_started":1}
I20260620 18:29:48.508330 4027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } }
I20260620 18:29:48.508659 4027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:48.508821 4027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:48.509029 4027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:48.511705 4027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } }
I20260620 18:29:48.512253 4027 leader_election.cc:302] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584f9fedda9844caa57536d0c301dae6; no voters:
I20260620 18:29:48.513010 4027 leader_election.cc:288] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:48.513214 4032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:48.513880 4032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [term 1 LEADER]: Becoming Leader. State: Replica: 584f9fedda9844caa57536d0c301dae6, State: Running, Role: LEADER
I20260620 18:29:48.514528 4032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } }
I20260620 18:29:48.515264 4027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:48.517553 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 584f9fedda9844caa57536d0c301dae6. Latest consensus state: current_term: 1 leader_uuid: "584f9fedda9844caa57536d0c301dae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } } }
I20260620 18:29:48.517966 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:48.518687 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "584f9fedda9844caa57536d0c301dae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584f9fedda9844caa57536d0c301dae6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38515 } } }
I20260620 18:29:48.519114 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:48.519486 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:48.523630 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:48.534763 4039 catalog_manager.cc:1347] Generated new cluster ID: 5a04243c7567414f8dba23d1917ab90c
I20260620 18:29:48.534873 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:48.557929 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:48.559577 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:48.568198 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 584f9fedda9844caa57536d0c301dae6: Generated new TSK 0
I20260620 18:29:48.569036 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:48.622921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:48.972410 4053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:48.972729 4053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:48.972924 4053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:48.980136 4053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:48.980238 4053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:48.980348 4053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:48.989842 4053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:48.991071 4053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:48.992347 4053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:49.006346 4058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:49.006255 4059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:49.006319 4061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:49.007306 4053 server_base.cc:1034] running on GCE node
I20260620 18:29:49.008021 4053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:49.009078 4053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:49.010288 4053 hybrid_clock.cc:648] HybridClock initialized: now 1781980189010215 us; error 64 us; skew 500 ppm
I20260620 18:29:49.010694 4053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:49 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:49.014158 4053 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:49.015800 4053 webserver.cc:460] Webserver started at http://127.0.97.1:34775/ using document root <none> and password file <none>
I20260620 18:29:49.016481 4053 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:49.016580 4053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:49.016815 4053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:49.019716 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "00373dd0518e428088c7605ebd2751f3"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "3e0f7c4c59898601a62f1df0ec60a0b8"
server_key_iv: "f26cc34cb17e42a32837a54fe29da4bb"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.020431 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "00373dd0518e428088c7605ebd2751f3"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "3e0f7c4c59898601a62f1df0ec60a0b8"
server_key_iv: "f26cc34cb17e42a32837a54fe29da4bb"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.025554 4053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:29:49.029028 4068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.030411 4053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:29:49.030566 4053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "00373dd0518e428088c7605ebd2751f3"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "3e0f7c4c59898601a62f1df0ec60a0b8"
server_key_iv: "f26cc34cb17e42a32837a54fe29da4bb"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.030735 4053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:49.060353 4053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:49.064523 4053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:49.064803 4053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:49.065721 4053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:49.067040 4053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:49.067174 4053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.067332 4053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:49.067413 4053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.100705 4053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44069
I20260620 18:29:49.100764 4180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44069 every 8 connection(s)
I20260620 18:29:49.101958 4053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:49.105686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4053
I20260620 18:29:49.105958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:49.106746 388 external_mini_cluster.cc:1428] Setting key 1425566673a3ac2b8c0537dac64a8a92
Jun 20 18:29:49 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:49.125788 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:49.104543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50369 (local address 127.0.97.62:38515)
0620 18:29:49.105024 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:49.105031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:49.106220 (+ 1189us) server_negotiation.cc:407] Connection header received
0620 18:29:49.108776 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:49.108780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:49.108851 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:49.108941 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:49.111116 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.111873 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.114407 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.115119 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.115266 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:49.119033 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:49.119042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:49.119049 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:49.119093 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:49.122657 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:49.123338 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:49.123347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:49.123353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:49.123503 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:49.123918 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:49.123926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:49.123931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:49.124319 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:49.124482 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:49.125143 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 18:29:49.125363 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260620 18:29:49.126695 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:49.105158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38515 (local address 127.0.97.1:50369)
0620 18:29:49.106029 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:49.106089 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:49.108485 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:49.109226 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:49.109237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:49.110082 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:49.110943 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:49.110973 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.112019 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.112028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:49.114190 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:49.114207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.115372 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.115377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:49.115643 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:49.116395 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:49.116418 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:49.118810 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:49.122812 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:49.122818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:49.122830 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:49.123221 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:49.123633 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:49.123636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:49.123639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:49.123809 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:49.124460 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:49.124465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:49.124766 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 18:29:49.125015 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:29:49.125462 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:29:49.128278 4181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38515
I20260620 18:29:49.128569 4181 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:49.129307 4181 heartbeater.cc:511] Master 127.0.97.62:38515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:29:49.131585 3992 ts_manager.cc:194] Registered new tserver with Master: 00373dd0518e428088c7605ebd2751f3 (127.0.97.1:44069)
I20260620 18:29:49.133147 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:49.154626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:49.475121 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:49.475381 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:49.475593 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:49.482888 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:49.482991 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:49.483186 4187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:49.492835 4187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:49.494021 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:49.495280 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:49.508566 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:49.508654 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:49.508682 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:49.509716 4187 server_base.cc:1034] running on GCE node
I20260620 18:29:49.510427 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:49.511422 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:49.512619 4187 hybrid_clock.cc:648] HybridClock initialized: now 1781980189512575 us; error 32 us; skew 500 ppm
I20260620 18:29:49.512926 4187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:49 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:49.516192 4187 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:49.517634 4187 webserver.cc:460] Webserver started at http://127.0.97.2:45405/ using document root <none> and password file <none>
I20260620 18:29:49.518298 4187 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:49.518443 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:49.518740 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:49.521605 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "1b26b67e293b43bfa3e80e9d2334ff3f"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "745f3fa866c7c92d71b1ea2dad811b11"
server_key_iv: "5a8e744db95f5fd453bdd5b357bd14ff"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.522373 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1b26b67e293b43bfa3e80e9d2334ff3f"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "745f3fa866c7c92d71b1ea2dad811b11"
server_key_iv: "5a8e744db95f5fd453bdd5b357bd14ff"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.527451 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 18:29:49.530790 4202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.532320 4187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:49.532490 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "1b26b67e293b43bfa3e80e9d2334ff3f"
format_stamp: "Formatted at 2026-06-20 18:29:49 on dist-test-slave-mcnm"
server_key: "745f3fa866c7c92d71b1ea2dad811b11"
server_key_iv: "5a8e744db95f5fd453bdd5b357bd14ff"
server_key_version: "encryptionkey@0"
I20260620 18:29:49.532686 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:49.554348 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:49.558497 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:49.558779 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:49.559876 4187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:49.561183 4187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:49.561257 4187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.561415 4187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:49.561492 4187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:49.594765 4187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32879
I20260620 18:29:49.594852 4314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32879 every 8 connection(s)
I20260620 18:29:49.596040 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:49.602589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4187
I20260620 18:29:49.602824 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:49.603381 388 external_mini_cluster.cc:1428] Setting key 5e7515824cede3075b9bc00787ab313b
Jun 20 18:29:49 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:49.620462 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:49.598774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60689 (local address 127.0.97.62:38515)
0620 18:29:49.598996 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:49.599001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:49.600437 (+ 1436us) server_negotiation.cc:407] Connection header received
0620 18:29:49.602483 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:49.602487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:49.602554 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:49.602643 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:49.604648 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.605620 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.608148 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.608566 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.608666 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:49.613292 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:49.613301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:49.613310 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:49.613354 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:49.617057 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:49.617830 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:49.617838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:49.617844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:49.618010 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:49.618451 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:49.618459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:49.618465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:49.618834 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:49.618999 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:49.619765 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 18:29:49.620022 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:29:49.621315 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:49.599220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38515 (local address 127.0.97.2:60689)
0620 18:29:49.600252 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:49.600297 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:49.602119 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:49.602936 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:49.602949 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:49.603696 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:49.604471 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:49.604496 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.605766 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.605771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:49.607985 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:49.608003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:49.608790 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:49.608799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:49.609128 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:49.609879 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:49.609913 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:49.613048 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:49.617211 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:49.617219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:49.617234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:49.617691 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:49.618123 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:49.618126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:49.618128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:49.618331 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:49.618972 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:49.618978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:49.619314 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 18:29:49.619631 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 18:29:49.620110 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":604,"threads_started":1}
I20260620 18:29:49.622751 4315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38515
I20260620 18:29:49.623061 4315 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:29:49.623809 4315 heartbeater.cc:511] Master 127.0.97.62:38515 requested a full tablet report, sending...
I20260620 18:29:49.625252 3992 ts_manager.cc:194] Registered new tserver with Master: 1b26b67e293b43bfa3e80e9d2334ff3f (127.0.97.2:32879)
I20260620 18:29:49.626125 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:49.650844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:49.969630 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:49.969902 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:49.969995 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:49.976881 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:49.976975 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:49.977082 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:49.986418 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:49.987655 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:49.988965 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:50.002185 4328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:50.002135 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:50.002100 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:50.002995 4320 server_base.cc:1034] running on GCE node
I20260620 18:29:50.003690 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:50.004647 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:50.005916 4320 hybrid_clock.cc:648] HybridClock initialized: now 1781980190005869 us; error 61 us; skew 500 ppm
I20260620 18:29:50.006242 4320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:50.009382 4320 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:50.010933 4320 webserver.cc:460] Webserver started at http://127.0.97.3:39459/ using document root <none> and password file <none>
I20260620 18:29:50.011700 4320 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:50.011852 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:50.012157 4320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:50.015065 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "e1486fb0b0e74f24b005416f20dd43e5"
format_stamp: "Formatted at 2026-06-20 18:29:50 on dist-test-slave-mcnm"
server_key: "28d3eeca7431811fed033e59b973a048"
server_key_iv: "95c196faf7d1e4cf60931f7a21f926f5"
server_key_version: "encryptionkey@0"
I20260620 18:29:50.015942 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e1486fb0b0e74f24b005416f20dd43e5"
format_stamp: "Formatted at 2026-06-20 18:29:50 on dist-test-slave-mcnm"
server_key: "28d3eeca7431811fed033e59b973a048"
server_key_iv: "95c196faf7d1e4cf60931f7a21f926f5"
server_key_version: "encryptionkey@0"
I20260620 18:29:50.020923 4320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:50.024473 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:50.025878 4320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:50.026204 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "e1486fb0b0e74f24b005416f20dd43e5"
format_stamp: "Formatted at 2026-06-20 18:29:50 on dist-test-slave-mcnm"
server_key: "28d3eeca7431811fed033e59b973a048"
server_key_iv: "95c196faf7d1e4cf60931f7a21f926f5"
server_key_version: "encryptionkey@0"
I20260620 18:29:50.026432 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:50.041347 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:50.045315 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:50.045607 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:50.046526 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:50.047845 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:50.047957 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:50.048112 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:50.048188 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:50.080210 4320 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42877
I20260620 18:29:50.080261 4447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42877 every 8 connection(s)
I20260620 18:29:50.081467 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:50.088527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4320
I20260620 18:29:50.088723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:50.089131 388 external_mini_cluster.cc:1428] Setting key 02f9c4e05e1bab35c729147393598a62
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:50.102000 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.083918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38777 (local address 127.0.97.62:38515)
0620 18:29:50.084092 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.084096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.085268 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 18:29:50.087079 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.087085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.087170 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.087301 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.088868 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.089727 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.091867 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.092300 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.092396 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.095792 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.095797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.095800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.095834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.098970 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.099764 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.099769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.099773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.099927 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.100361 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.100366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.100372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.100614 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.100766 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.101431 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.101643 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:29:50.103103 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.084353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38515 (local address 127.0.97.3:38777)
0620 18:29:50.085096 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:50.085134 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:50.086763 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:50.087595 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:50.087604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:50.088103 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:50.088642 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:50.088692 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.089912 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.089917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:50.091703 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:50.091716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.092483 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.092487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:50.092672 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.093303 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:50.093324 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:50.095555 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:50.099123 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:50.099132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:50.099148 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:50.099643 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:50.100035 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:50.100043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:50.100048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:50.100235 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:50.100740 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:50.100749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:50.101067 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:29:50.101330 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 18:29:50.101812 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260620 18:29:50.104558 4448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38515
I20260620 18:29:50.104873 4448 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:50.105756 4448 heartbeater.cc:511] Master 127.0.97.62:38515 requested a full tablet report, sending...
I20260620 18:29:50.107213 3992 ts_manager.cc:194] Registered new tserver with Master: e1486fb0b0e74f24b005416f20dd43e5 (127.0.97.3:42877)
I20260620 18:29:50.108031 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38777
I20260620 18:29:50.117130 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:50.136595 4181 heartbeater.cc:503] Master 127.0.97.62:38515 was elected leader, sending a full tablet report...
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:50.167722 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.150612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.97.1:44069)
0620 18:29:50.151262 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.151297 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.151331 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:29:50.151589 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.151596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.151830 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.152029 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.153425 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.154543 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.156760 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.157817 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.157958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.162141 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.162151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.162165 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.162203 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.165137 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.165732 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.165739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.165741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.165868 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.166181 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.166185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.166187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.166522 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.166730 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.167089 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.167278 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":311,"threads_started":1}
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:50.185074 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.171802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.97.62:38515)
0620 18:29:50.172020 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.172027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.172227 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:29:50.172415 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.172421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.172496 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.172605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.173768 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.174463 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.175838 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.176240 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.176370 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.180033 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.180038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.180042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.180072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.182636 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.183265 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.183268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.183271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.183395 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.183808 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.183811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.183813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.184118 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.184292 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.184634 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.184832 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:29:50.185678 3992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42052
I20260620 18:29:50.186720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4053
I20260620 18:29:50.199903 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.187796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.97.1:44069)
0620 18:29:50.188003 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.188009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.188201 (+ 192us) server_negotiation.cc:407] Connection header received
0620 18:29:50.188395 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.188399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.188524 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.188634 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.190257 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.190930 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.192216 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.192626 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.192753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.194826 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.194831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.194840 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.194870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.197324 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.198089 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.198095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.198099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.198218 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.198581 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.198587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.198590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.198825 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.199082 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.199364 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.199597 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:29:50.200537 4176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:50.281678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4053
I20260620 18:29:50.298595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4187
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:50.314162 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.299562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.0.97.2:32879)
0620 18:29:50.300044 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.300052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.300077 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:29:50.300141 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.300147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.300374 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.300508 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.301650 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.302647 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.303974 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.304534 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.304648 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.308557 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.308567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.308581 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.308612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.311415 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.312247 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.312254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.312256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.312400 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.312734 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.312737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.312739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.313067 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.313268 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.313563 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.313828 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":256,"threads_started":1}
I20260620 18:29:50.315413 4310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:50.397255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4187
I20260620 18:29:50.412858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4320
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:50.430258 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:50.413874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.97.3:42877)
0620 18:29:50.414540 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:50.414547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:50.414577 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:29:50.414634 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:50.414640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:50.414848 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:50.415056 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:50.416507 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.417553 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.418859 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:50.419426 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:50.419642 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:50.424114 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:50.424126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:50.424140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:50.424174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:50.427178 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.427899 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.427906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.427908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.428021 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:50.428485 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:50.428489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:50.428492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:50.428858 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:50.429144 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:50.429620 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 18:29:50.429904 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":341,"threads_started":1}
I20260620 18:29:50.431725 4443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:50.512880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4320
I20260620 18:29:50.530201 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 3960
I20260620 18:29:50.531489 4022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:50.632529 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 3960
2026-06-20T18:29:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3852 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[4467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[4467](info): set up 2 sockets
Jun 20 18:29:50 dist-test-slave-mcnm krb5kdc[4467](info): commencing operation
krb5kdc: starting...
W20260620 18:29:51.735929 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:29:51 dist-test-slave-mcnm krb5kdc[4467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:29:51.935096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:52.251184 4480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:52.251524 4480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:52.251726 4480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:52.258746 4480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:52.258898 4480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:52.259004 4480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:52.259110 4480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:52.259200 4480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:52.268751 4480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:52.270040 4480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:52.271324 4480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:52.282658 4485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:52.282693 4486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:52.282864 4488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:52.283504 4480 server_base.cc:1034] running on GCE node
I20260620 18:29:52.284298 4480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:52.285640 4480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:52.286875 4480 hybrid_clock.cc:648] HybridClock initialized: now 1781980192286842 us; error 51 us; skew 500 ppm
I20260620 18:29:52.287217 4480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:52 dist-test-slave-mcnm krb5kdc[4467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:52.290386 4480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:29:52.291617 4480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:52.291718 4480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:52.291975 4480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:52.294821 4480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "930119331159447b965ad89bb6817908"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "328a0ecf53da96bd59ff734d4147ac96"
server_key_iv: "18223eb3f3ca473b7ab2ee004a1ae463"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.295599 4480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "930119331159447b965ad89bb6817908"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "328a0ecf53da96bd59ff734d4147ac96"
server_key_iv: "18223eb3f3ca473b7ab2ee004a1ae463"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.300452 4480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 18:29:52.303877 4494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:52.305161 4480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:29:52.305300 4480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "930119331159447b965ad89bb6817908"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "328a0ecf53da96bd59ff734d4147ac96"
server_key_iv: "18223eb3f3ca473b7ab2ee004a1ae463"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.305472 4480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:52.325788 4480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:52.330047 4480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:52.330271 4480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:52.348721 4480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32989
I20260620 18:29:52.348702 4545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32989 every 8 connection(s)
I20260620 18:29:52.349612 4480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:52.352675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4480
I20260620 18:29:52.352890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:52.353242 4546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:52.353570 388 external_mini_cluster.cc:1428] Setting key 18a024e579f0bc9773d559676b6d86bc
I20260620 18:29:52.365228 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: Bootstrap starting.
Jun 20 18:29:52 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:52.368858 4546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:52.370028 4546 log.cc:824] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:52.373435 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: No bootstrap required, opened a new log
I20260620 18:29:52.375483 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:52.355467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.0.97.62:32989)
0620 18:29:52.356312 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:52.356324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:52.356369 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:29:52.357798 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:52.357819 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:52.358183 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:52.358586 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:52.360124 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.361355 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.362624 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.363330 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.363597 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:52.367918 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:52.367926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:52.367939 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:52.367977 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:52.371379 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:52.372266 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:52.372272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:52.372277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:52.372434 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:52.372858 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:52.372861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:52.372864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:52.373341 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:52.373604 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:52.374055 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:29:52.374458 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":400,"threads_started":1}
I20260620 18:29:52.382153 4546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } }
I20260620 18:29:52.382649 4546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:52.382771 4546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930119331159447b965ad89bb6817908, State: Initialized, Role: FOLLOWER
I20260620 18:29:52.383355 4546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } }
I20260620 18:29:52.383630 4546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:52.383802 4546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:52.383985 4546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:52.386216 4546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } }
I20260620 18:29:52.386858 4546 leader_election.cc:302] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 930119331159447b965ad89bb6817908; no voters:
I20260620 18:29:52.387439 4546 leader_election.cc:288] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:52.387600 4551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:52.388337 4551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [term 1 LEADER]: Becoming Leader. State: Replica: 930119331159447b965ad89bb6817908, State: Running, Role: LEADER
I20260620 18:29:52.388859 4551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } }
I20260620 18:29:52.389732 4546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:52.391989 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "930119331159447b965ad89bb6817908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } } }
I20260620 18:29:52.392299 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:52.391891 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 930119331159447b965ad89bb6817908. Latest consensus state: current_term: 1 leader_uuid: "930119331159447b965ad89bb6817908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930119331159447b965ad89bb6817908" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32989 } } }
I20260620 18:29:52.393034 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:52.395712 4557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:52.400728 4557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:52.409231 4557 catalog_manager.cc:1347] Generated new cluster ID: ae3b7b4523b9419f86ce14c3c6476f4e
I20260620 18:29:52.409371 4557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:52.421969 4557 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:52.423249 4557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:52.429680 4557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: Generated new TSK 0
I20260620 18:29:52.430526 4557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:29:52.491737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:52.837581 4572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:52.837874 4572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:52.837971 4572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:52.845268 4572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:52.845378 4572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:52.845494 4572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:52.855178 4572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:52.856506 4572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:52.857765 4572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:52.870800 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:52.870858 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:52.870854 4577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:52.871726 4572 server_base.cc:1034] running on GCE node
I20260620 18:29:52.872350 4572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:52.873273 4572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:52.874506 4572 hybrid_clock.cc:648] HybridClock initialized: now 1781980192874423 us; error 79 us; skew 500 ppm
I20260620 18:29:52.874843 4572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:52 dist-test-slave-mcnm krb5kdc[4467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:52.878021 4572 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:29:52.879091 4572 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:52.879204 4572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:52.879595 4572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:52.882562 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "0dee75bddd3946beac3e7f1dc3232c1d"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "326e29be36af4c33acaa3eaa47ba3f5f"
server_key_iv: "01ecd108f57ad3e08fd1701643e07df2"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.883399 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0dee75bddd3946beac3e7f1dc3232c1d"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "326e29be36af4c33acaa3eaa47ba3f5f"
server_key_iv: "01ecd108f57ad3e08fd1701643e07df2"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.888578 4572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:52.892140 4586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:52.893610 4572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:29:52.893795 4572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "0dee75bddd3946beac3e7f1dc3232c1d"
format_stamp: "Formatted at 2026-06-20 18:29:52 on dist-test-slave-mcnm"
server_key: "326e29be36af4c33acaa3eaa47ba3f5f"
server_key_iv: "01ecd108f57ad3e08fd1701643e07df2"
server_key_version: "encryptionkey@0"
I20260620 18:29:52.894006 4572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:52.910279 4572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:52.914703 4572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:52.914984 4572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:52.916029 4572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:52.917456 4572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:52.917554 4572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:52.917682 4572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:52.917762 4572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:52.952447 4572 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38463
I20260620 18:29:52.952541 4698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38463 every 8 connection(s)
I20260620 18:29:52.953320 4572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:52.963744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4572
I20260620 18:29:52.963968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:52.964677 388 external_mini_cluster.cc:1428] Setting key 184403941c856619868014806d901575
Jun 20 18:29:52 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:52.973465 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:52.955871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58493 (local address 127.0.97.62:32989)
0620 18:29:52.956268 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:52.956273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:52.957383 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 18:29:52.959144 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:52.959149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:52.959225 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:52.959324 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:52.960918 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.961657 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.963317 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.964007 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.964189 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:52.967842 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:52.967847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:52.967851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:52.967881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:52.970398 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:52.971120 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:52.971128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:52.971132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:52.971253 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:52.971742 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:52.971749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:52.971753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:52.972012 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:52.972172 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:52.972882 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 18:29:52.973102 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":188,"threads_started":1}
I20260620 18:29:52.974560 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:52.956501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32989 (local address 127.0.97.1:58493)
0620 18:29:52.957211 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:52.957253 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:52.958876 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:52.959584 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:52.959615 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:52.960138 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:52.960732 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:52.960753 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.961821 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.961826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:52.963152 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:52.963164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:52.964255 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:52.964261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:52.964448 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:52.965138 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:52.965160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:52.967631 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:52.970550 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:52.970560 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:52.970578 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:52.970999 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:52.971370 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:52.971375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:52.971379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:52.971636 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:52.972144 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:52.972152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:52.972504 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:29:52.972783 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 18:29:52.973318 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":284,"threads_started":1}
I20260620 18:29:52.976217 4699 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32989
I20260620 18:29:52.976545 4699 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:52.977207 4699 heartbeater.cc:511] Master 127.0.97.62:32989 requested a full tablet report, sending...
I20260620 18:29:52.979561 4511 ts_manager.cc:194] Registered new tserver with Master: 0dee75bddd3946beac3e7f1dc3232c1d (127.0.97.1:38463)
I20260620 18:29:52.981679 4511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58493
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:29:53.011006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:53.332559 4705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:53.332828 4705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:53.332922 4705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:53.340094 4705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:53.340200 4705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:53.340312 4705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:53.350018 4705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:53.351279 4705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:53.352610 4705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:53.365799 4710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:53.365760 4713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:53.365717 4711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:53.366350 4705 server_base.cc:1034] running on GCE node
I20260620 18:29:53.366932 4705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:53.367908 4705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:53.369189 4705 hybrid_clock.cc:648] HybridClock initialized: now 1781980193369141 us; error 66 us; skew 500 ppm
I20260620 18:29:53.369455 4705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:53 dist-test-slave-mcnm krb5kdc[4467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:53.372357 4705 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:29:53.373482 4705 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:53.373621 4705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:53.373930 4705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:53.376770 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "0ed327714ec09f96a4cc95c4420ed401"
server_key_iv: "6c15fc11ace95ce3477752a24d4d668d"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.377544 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "0ed327714ec09f96a4cc95c4420ed401"
server_key_iv: "6c15fc11ace95ce3477752a24d4d668d"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.382377 4705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:29:53.385766 4719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.387074 4705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:29:53.387254 4705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "0ed327714ec09f96a4cc95c4420ed401"
server_key_iv: "6c15fc11ace95ce3477752a24d4d668d"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.387481 4705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:53.410766 4705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:53.418401 4705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:53.418638 4705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:53.419689 4705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:53.420948 4705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:53.421033 4705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.421109 4705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:53.421162 4705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.453384 4705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46817
I20260620 18:29:53.453433 4831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46817 every 8 connection(s)
I20260620 18:29:53.454231 4705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:53.459787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4705
I20260620 18:29:53.459976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:53.460439 388 external_mini_cluster.cc:1428] Setting key 24f90d5b64eab5bc8ee6bfee6824fe2b
Jun 20 18:29:53 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:53.474778 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:53.456776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53007 (local address 127.0.97.62:32989)
0620 18:29:53.457060 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:53.457068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:53.458229 (+ 1161us) server_negotiation.cc:407] Connection header received
0620 18:29:53.460419 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:53.460426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:53.460569 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:53.460683 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:53.462298 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.462996 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.464593 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.465021 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.465153 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:53.468619 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:53.468628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:53.468636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:53.468679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:53.471367 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.472338 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.472343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.472345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.472496 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.472973 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.472976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.472979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.473265 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:53.473381 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:53.474275 (+ 894us) server_negotiation.cc:299] Negotiation successful
0620 18:29:53.474439 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:29:53.476306 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:53.457309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32989 (local address 127.0.97.2:53007)
0620 18:29:53.458049 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:53.458091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:53.460149 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:53.460851 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:53.460859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:53.461507 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:53.462120 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:53.462145 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.463147 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.463152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:53.464451 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:53.464462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.465240 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.465247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:53.465475 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:53.466066 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:53.466087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:53.468364 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:53.471503 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:53.471538 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:53.471558 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:53.472184 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:53.472616 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:53.472623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:53.472630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:53.472848 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:53.473400 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:53.473411 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:53.473845 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 18:29:53.474155 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 18:29:53.474716 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
I20260620 18:29:53.477769 4832 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32989
I20260620 18:29:53.478129 4832 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:53.478863 4832 heartbeater.cc:511] Master 127.0.97.62:32989 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:29:53.480255 4511 ts_manager.cc:194] Registered new tserver with Master: 8e50dc37dd4e4f9fba17f60b055f2cfe (127.0.97.2:46817)
I20260620 18:29:53.481097 4511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:29:53.507468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:29:53.826380 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:53.826714 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:53.826818 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:53.834025 4837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:53.834137 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:53.834254 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:53.844036 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:53.845292 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:53.846608 4837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:53.859411 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:53.859442 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:53.859493 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:53.860785 4837 server_base.cc:1034] running on GCE node
I20260620 18:29:53.861443 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:53.862421 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:53.863688 4837 hybrid_clock.cc:648] HybridClock initialized: now 1781980193863614 us; error 60 us; skew 500 ppm
I20260620 18:29:53.864050 4837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:29:53 dist-test-slave-mcnm krb5kdc[4467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:29:53.867489 4837 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:29:53.868810 4837 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:53.868952 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:53.869280 4837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:53.872298 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "beca36bf82a4441a9a834bad8ebea5a7"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "b839ca731dac99ff1caecc0c8c237229"
server_key_iv: "2689c051ad5f44c435c76ed047b98fc7"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.873152 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "beca36bf82a4441a9a834bad8ebea5a7"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "b839ca731dac99ff1caecc0c8c237229"
server_key_iv: "2689c051ad5f44c435c76ed047b98fc7"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.878486 4837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:29:53.882115 4851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.883766 4837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:53.883965 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "beca36bf82a4441a9a834bad8ebea5a7"
format_stamp: "Formatted at 2026-06-20 18:29:53 on dist-test-slave-mcnm"
server_key: "b839ca731dac99ff1caecc0c8c237229"
server_key_iv: "2689c051ad5f44c435c76ed047b98fc7"
server_key_version: "encryptionkey@0"
I20260620 18:29:53.884387 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:53.901932 4837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:53.906647 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:53.906937 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:53.908000 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:53.909442 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:53.909566 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.909698 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:53.909795 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:53.942466 4837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36657
I20260620 18:29:53.942512 4963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36657 every 8 connection(s)
I20260620 18:29:53.943316 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:53.945858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 4837
I20260620 18:29:53.946102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:53.946506 388 external_mini_cluster.cc:1428] Setting key 9213e0593786b3d53684e626a6095803
Jun 20 18:29:53 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:29:53.963418 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:53.945928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47301 (local address 127.0.97.62:32989)
0620 18:29:53.946119 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:53.946124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:53.947412 (+ 1288us) server_negotiation.cc:407] Connection header received
0620 18:29:53.949359 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:53.949364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:53.949463 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:53.949567 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:53.951025 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.951837 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.953368 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.953786 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.953916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:53.957557 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:53.957563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:53.957566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:53.957596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:53.960133 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.961049 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.961055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.961060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.961218 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.961730 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.961735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.961738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.962086 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:53.962199 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:53.963020 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 18:29:53.963177 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:29:53.964425 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:53.946421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32989 (local address 127.0.97.3:47301)
0620 18:29:53.947258 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:53.947311 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:53.949075 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:53.949800 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:53.949808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:53.950292 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:53.950846 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:53.950863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.951978 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.951983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:53.953222 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:53.953234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.954026 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.954033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:53.954240 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:53.954911 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:29:53.954937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:53.957300 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:53.960291 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:53.960301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:53.960317 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:53.960905 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:53.961354 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:29:53.961359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:29:53.961362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:29:53.961587 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:29:53.962295 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:53.962301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:53.962648 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:29:53.962895 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 18:29:53.963283 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":405,"threads_started":1}
I20260620 18:29:53.965726 4964 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32989
I20260620 18:29:53.966041 4964 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:53.966706 4964 heartbeater.cc:511] Master 127.0.97.62:32989 requested a full tablet report, sending...
I20260620 18:29:53.968055 4511 ts_manager.cc:194] Registered new tserver with Master: beca36bf82a4441a9a834bad8ebea5a7 (127.0.97.3:36657)
I20260620 18:29:53.968837 4511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47301
I20260620 18:29:53.973855 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:53.984802 4699 heartbeater.cc:503] Master 127.0.97.62:32989 was elected leader, sending a full tablet report...
I20260620 18:29:53.991006 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:53.978236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.97.62:32989)
0620 18:29:53.978390 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:53.978394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:53.978820 (+ 426us) server_negotiation.cc:407] Connection header received
0620 18:29:53.978944 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:53.978947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:53.979007 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:53.979088 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:53.980552 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.981235 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.982392 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:53.982795 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:53.982887 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:53.985260 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:53.985265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:53.985269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:53.985299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:53.988280 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.988943 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.988947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.988950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.989132 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:53.989514 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:53.989521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:53.989527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:53.989907 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:53.990154 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:53.990487 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 18:29:53.990716 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:29:53.995201 4511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:54.002220 4511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:54.031117 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.015794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.0.97.1:38463)
0620 18:29:54.016499 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.016511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.017474 (+ 963us) server_negotiation.cc:407] Connection header received
0620 18:29:54.017642 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.017653 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.017923 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.019486 (+ 1563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:54.020761 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.024293 (+ 3532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.028879 (+ 4586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.029955 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.030124 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.030365 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.030589 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.030753 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":366,"threads_started":1}
I20260620 18:29:54.031226 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.015044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38463 (local address 127.0.0.1:46552)
0620 18:29:54.017359 (+ 2315us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:54.017371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:54.017500 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:54.019647 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:54.019659 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:54.019697 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:54.020285 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.020302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.024540 (+ 4238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.024549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.028715 (+ 4166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.028726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.030144 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.030150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.030343 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.030396 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:29:54.030537 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:29:54.030746 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"mutex_wait_us":49,"thread_start_us":1380,"threads_started":1}
I20260620 18:29:54.035358 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.014702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36657 (local address 127.0.0.1:34072)
0620 18:29:54.016799 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:54.016830 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:54.016960 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:54.028195 (+ 11235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:54.028200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:54.028226 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:54.028692 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.028707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.030596 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.030602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.032805 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.032819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.034593 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.034600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.034756 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.034790 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:29:54.034889 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:29:54.035025 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1908,"mutex_wait_us":19,"thread_start_us":2446,"threads_started":1}
I20260620 18:29:54.035421 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.015568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.97.3:36657)
0620 18:29:54.027502 (+ 11934us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.027531 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.027575 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:29:54.027684 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.027693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.027978 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.028183 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:54.028922 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.030443 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.032977 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.034387 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.034523 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.034734 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.034941 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.035107 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11765,"thread_start_us":11625,"threads_started":1}
I20260620 18:29:54.035275 4634 tablet_service.cc:1467] Processing CreateTablet for tablet e4668d97f7384a2eb333cc3949d33b12 (DEFAULT_TABLE table=test-table [id=5e017c975d6b45f6b1a8f47d412eb002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:54.036469 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.017547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46817 (local address 127.0.0.1:55998)
0620 18:29:54.026752 (+ 9205us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:54.026777 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:54.026920 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:54.028541 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:54.028546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:54.028585 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:54.029083 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.029096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.030863 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.030868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.033631 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.033641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.035685 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.035692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.035853 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.035883 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:29:54.035982 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:29:54.036116 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9019,"mutex_wait_us":43,"thread_start_us":9866,"threads_started":1}
I20260620 18:29:54.036577 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.017818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.97.2:46817)
0620 18:29:54.027969 (+ 10151us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.027977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.028004 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:29:54.028109 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.028118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.028335 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.028549 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:54.029260 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.030703 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.033787 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.034803 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.034916 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.035091 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.036066 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.036241 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10004,"thread_start_us":9886,"threads_started":1}
I20260620 18:29:54.037856 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4668d97f7384a2eb333cc3949d33b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:54.038986 4899 tablet_service.cc:1467] Processing CreateTablet for tablet e4668d97f7384a2eb333cc3949d33b12 (DEFAULT_TABLE table=test-table [id=5e017c975d6b45f6b1a8f47d412eb002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:54.039394 4767 tablet_service.cc:1467] Processing CreateTablet for tablet e4668d97f7384a2eb333cc3949d33b12 (DEFAULT_TABLE table=test-table [id=5e017c975d6b45f6b1a8f47d412eb002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:54.040352 4899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4668d97f7384a2eb333cc3949d33b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:54.040477 4767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4668d97f7384a2eb333cc3949d33b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:54.050886 4982 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Bootstrap starting.
I20260620 18:29:54.054261 4982 tablet_bootstrap.cc:654] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:54.055240 4982 log.cc:824] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:54.057909 4982 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: No bootstrap required, opened a new log
I20260620 18:29:54.058171 4982 ts_tablet_manager.cc:1397] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:29:54.059795 4984 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Bootstrap starting.
I20260620 18:29:54.060636 4983 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Bootstrap starting.
I20260620 18:29:54.062906 4984 tablet_bootstrap.cc:654] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:54.064007 4983 tablet_bootstrap.cc:654] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:54.064257 4984 log.cc:824] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:54.065227 4983 log.cc:824] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:54.067365 4984 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: No bootstrap required, opened a new log
I20260620 18:29:54.067680 4984 ts_tablet_manager.cc:1397] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 18:29:54.067754 4982 raft_consensus.cc:348] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.068192 4983 tablet_bootstrap.cc:492] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: No bootstrap required, opened a new log
I20260620 18:29:54.068204 4982 raft_consensus.cc:374] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:54.068306 4982 raft_consensus.cc:729] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e50dc37dd4e4f9fba17f60b055f2cfe, State: Initialized, Role: FOLLOWER
I20260620 18:29:54.068490 4983 ts_tablet_manager.cc:1397] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:29:54.068938 4982 consensus_queue.cc:260] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.070034 4832 heartbeater.cc:503] Master 127.0.97.62:32989 was elected leader, sending a full tablet report...
I20260620 18:29:54.070664 4982 ts_tablet_manager.cc:1428] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 18:29:54.077450 4983 raft_consensus.cc:348] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.077811 4983 raft_consensus.cc:374] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:54.077926 4983 raft_consensus.cc:729] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dee75bddd3946beac3e7f1dc3232c1d, State: Initialized, Role: FOLLOWER
I20260620 18:29:54.078215 4984 raft_consensus.cc:348] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.078613 4984 raft_consensus.cc:374] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:54.078492 4983 consensus_queue.cc:260] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.078723 4984 raft_consensus.cc:729] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beca36bf82a4441a9a834bad8ebea5a7, State: Initialized, Role: FOLLOWER
I20260620 18:29:54.079254 4984 consensus_queue.cc:260] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.080302 4964 heartbeater.cc:503] Master 127.0.97.62:32989 was elected leader, sending a full tablet report...
I20260620 18:29:54.080971 4984 ts_tablet_manager.cc:1428] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260620 18:29:54.081374 4983 ts_tablet_manager.cc:1428] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260620 18:29:54.194851 4965 tablet.cc:2369] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:54.205585 4700 tablet.cc:2369] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:29:54.206080 4833 tablet.cc:2369] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:54.396639 4989 raft_consensus.cc:482] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:54.397037 4989 raft_consensus.cc:504] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.398828 4989 leader_election.cc:288] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beca36bf82a4441a9a834bad8ebea5a7 (127.0.97.3:36657), 8e50dc37dd4e4f9fba17f60b055f2cfe (127.0.97.2:46817)
I20260620 18:29:54.405411 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.399284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54749 (local address 127.0.97.3:36657)
0620 18:29:54.399566 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.399575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.400144 (+ 569us) server_negotiation.cc:407] Connection header received
0620 18:29:54.400380 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.400387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.400490 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.400596 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:54.401163 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.401875 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.403689 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.404474 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.404637 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.404746 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.405025 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.405150 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:29:54.405508 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.399283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36657 (local address 127.0.97.1:54749)
0620 18:29:54.400003 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:54.400020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:54.400220 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:54.400619 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:54.400625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:54.400673 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:54.401004 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.401014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.402019 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.402026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.403476 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.403487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.404693 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.404699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.404866 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.404890 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:29:54.405008 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:29:54.405136 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":421,"threads_started":1}
I20260620 18:29:54.406006 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.400259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60807 (local address 127.0.97.2:46817)
0620 18:29:54.400503 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.400510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.400922 (+ 412us) server_negotiation.cc:407] Connection header received
0620 18:29:54.401127 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.401131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.401187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.401284 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:29:54.401849 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.402535 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.404310 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.405074 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.405232 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.405329 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.405667 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.405788 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:29:54.406522 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.400151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46817 (local address 127.0.97.1:60807)
0620 18:29:54.400814 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:54.400828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:54.400984 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:54.401295 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:54.401300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:54.401320 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:29:54.401685 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.401693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.402706 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.402711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.404152 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:54.404163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.405304 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.405310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:54.405456 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.405503 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:29:54.406213 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 18:29:54.406302 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":329,"threads_started":1}
I20260620 18:29:54.406513 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4668d97f7384a2eb333cc3949d33b12" candidate_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beca36bf82a4441a9a834bad8ebea5a7" is_pre_election: true
I20260620 18:29:54.407053 4919 raft_consensus.cc:2454] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dee75bddd3946beac3e7f1dc3232c1d in term 0.
I20260620 18:29:54.407768 4589 leader_election.cc:302] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dee75bddd3946beac3e7f1dc3232c1d, beca36bf82a4441a9a834bad8ebea5a7; no voters:
I20260620 18:29:54.407758 4787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4668d97f7384a2eb333cc3949d33b12" candidate_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" is_pre_election: true
I20260620 18:29:54.408243 4989 raft_consensus.cc:2781] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:54.408277 4787 raft_consensus.cc:2454] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dee75bddd3946beac3e7f1dc3232c1d in term 0.
I20260620 18:29:54.408396 4989 raft_consensus.cc:482] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:54.408505 4989 raft_consensus.cc:3037] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:54.410826 4989 raft_consensus.cc:504] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.411386 4989 leader_election.cc:288] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [CANDIDATE]: Term 1 election: Requested vote from peers beca36bf82a4441a9a834bad8ebea5a7 (127.0.97.3:36657), 8e50dc37dd4e4f9fba17f60b055f2cfe (127.0.97.2:46817)
I20260620 18:29:54.412029 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4668d97f7384a2eb333cc3949d33b12" candidate_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beca36bf82a4441a9a834bad8ebea5a7"
I20260620 18:29:54.412029 4787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4668d97f7384a2eb333cc3949d33b12" candidate_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe"
I20260620 18:29:54.412240 4787 raft_consensus.cc:3037] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:54.412245 4919 raft_consensus.cc:3037] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:54.415405 4787 raft_consensus.cc:2454] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dee75bddd3946beac3e7f1dc3232c1d in term 1.
I20260620 18:29:54.415405 4919 raft_consensus.cc:2454] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dee75bddd3946beac3e7f1dc3232c1d in term 1.
I20260620 18:29:54.415902 4589 leader_election.cc:302] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dee75bddd3946beac3e7f1dc3232c1d, beca36bf82a4441a9a834bad8ebea5a7; no voters:
I20260620 18:29:54.416218 4989 raft_consensus.cc:2781] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:54.416676 4989 raft_consensus.cc:686] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 1 LEADER]: Becoming Leader. State: Replica: 0dee75bddd3946beac3e7f1dc3232c1d, State: Running, Role: LEADER
I20260620 18:29:54.417178 4989 consensus_queue.cc:237] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } }
I20260620 18:29:54.422156 4510 catalog_manager.cc:5495] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dee75bddd3946beac3e7f1dc3232c1d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dee75bddd3946beac3e7f1dc3232c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260620 18:29:54.479354 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.474513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.97.1:38463)
0620 18:29:54.474688 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.474692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.474708 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:29:54.474792 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.474795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.474876 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.475002 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:29:54.475348 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.476149 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.477494 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.477873 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.477986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.478422 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:29:54.478913 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:29:54.479037 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.479099 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.479157 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:29:54.487438 4919 raft_consensus.cc:1261] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 1 FOLLOWER]: Refusing update from remote peer 0dee75bddd3946beac3e7f1dc3232c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:54.487572 4787 raft_consensus.cc:1261] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 1 FOLLOWER]: Refusing update from remote peer 0dee75bddd3946beac3e7f1dc3232c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:54.488512 4993 consensus_queue.cc:1035] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "beca36bf82a4441a9a834bad8ebea5a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:54.488873 4989 consensus_queue.cc:1035] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e50dc37dd4e4f9fba17f60b055f2cfe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:54.515118 4511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:54.515412 4511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:54.518666 4511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 930119331159447b965ad89bb6817908: Sending DeleteTablet for 3 replicas of tablet e4668d97f7384a2eb333cc3949d33b12
I20260620 18:29:54.519901 4899 tablet_service.cc:1514] Processing DeleteTablet for tablet e4668d97f7384a2eb333cc3949d33b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34072
I20260620 18:29:54.519915 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet e4668d97f7384a2eb333cc3949d33b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46552
I20260620 18:29:54.520119 4767 tablet_service.cc:1514] Processing DeleteTablet for tablet e4668d97f7384a2eb333cc3949d33b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55998
I20260620 18:29:54.523666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4572
I20260620 18:29:54.532505 5008 tablet_replica.cc:330] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: stopping tablet replica
I20260620 18:29:54.533159 5008 raft_consensus.cc:2229] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:54.533803 5008 raft_consensus.cc:2258] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:54.535423 5006 tablet_replica.cc:330] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: stopping tablet replica
I20260620 18:29:54.535982 5006 raft_consensus.cc:2229] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:54.536988 5006 raft_consensus.cc:2258] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:54.537606 5007 tablet_replica.cc:330] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: stopping tablet replica
I20260620 18:29:54.538311 5007 raft_consensus.cc:2229] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:54.538870 5007 raft_consensus.cc:2258] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:54.540313 5006 ts_tablet_manager.cc:1905] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:54.541708 5007 ts_tablet_manager.cc:1905] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:54.542758 5008 ts_tablet_manager.cc:1905] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:29:54 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:29:54.548460 5006 ts_tablet_manager.cc:1918] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:54.548664 5006 log.cc:1196] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/e4668d97f7384a2eb333cc3949d33b12
I20260620 18:29:54.549288 5006 ts_tablet_manager.cc:1939] T e4668d97f7384a2eb333cc3949d33b12 P 0dee75bddd3946beac3e7f1dc3232c1d: Deleting consensus metadata
I20260620 18:29:54.551267 4496 catalog_manager.cc:4841] TS 0dee75bddd3946beac3e7f1dc3232c1d (127.0.97.1:38463): tablet e4668d97f7384a2eb333cc3949d33b12 (table test-table [id=5e017c975d6b45f6b1a8f47d412eb002]) successfully deleted
I20260620 18:29:54.554699 5008 ts_tablet_manager.cc:1918] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:54.554805 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.525611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.0.97.1:38463)
0620 18:29:54.525840 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.525851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.534734 (+ 8883us) server_negotiation.cc:407] Connection header received
0620 18:29:54.534946 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.534955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.535078 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.535248 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:54.537675 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.538835 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.540710 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.541437 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.541554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.547026 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:54.547042 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:54.547062 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:54.547111 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:54.551373 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.552256 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.552264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.552267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.552425 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.552864 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.552870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.552875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.553278 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:54.553595 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.554149 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.554390 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:29:54.554881 5008 log.cc:1196] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/e4668d97f7384a2eb333cc3949d33b12
I20260620 18:29:54.555446 5008 ts_tablet_manager.cc:1939] T e4668d97f7384a2eb333cc3949d33b12 P 8e50dc37dd4e4f9fba17f60b055f2cfe: Deleting consensus metadata
I20260620 18:29:54.555593 5007 ts_tablet_manager.cc:1918] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:54.555799 5007 log.cc:1196] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/e4668d97f7384a2eb333cc3949d33b12
I20260620 18:29:54.556473 4694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:54.557246 4497 catalog_manager.cc:4841] TS 8e50dc37dd4e4f9fba17f60b055f2cfe (127.0.97.2:46817): tablet e4668d97f7384a2eb333cc3949d33b12 (table test-table [id=5e017c975d6b45f6b1a8f47d412eb002]) successfully deleted
I20260620 18:29:54.557447 5007 ts_tablet_manager.cc:1939] T e4668d97f7384a2eb333cc3949d33b12 P beca36bf82a4441a9a834bad8ebea5a7: Deleting consensus metadata
I20260620 18:29:54.559422 4497 catalog_manager.cc:4841] TS beca36bf82a4441a9a834bad8ebea5a7 (127.0.97.3:36657): tablet e4668d97f7384a2eb333cc3949d33b12 (table test-table [id=5e017c975d6b45f6b1a8f47d412eb002]) successfully deleted
I20260620 18:29:54.654919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4572
I20260620 18:29:54.669155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4705
Jun 20 18:29:54 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:29:54.684005 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.669989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.0.97.2:46817)
0620 18:29:54.670168 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.670176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.670195 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:29:54.670402 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.670407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.670483 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.670590 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:54.671840 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.672559 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.673758 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.674160 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.674290 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.678443 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:54.678456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:54.678471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:54.678506 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:54.681319 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.681959 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.681966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.681969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.682108 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.682480 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.682486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.682490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.682846 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:54.683137 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.683422 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.683674 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:29:54.684657 4827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:54.781260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4705
I20260620 18:29:54.800294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4837
Jun 20 18:29:54 dist-test-slave-mcnm krb5kdc[4467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:29:54.815249 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:54.801209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.97.3:36657)
0620 18:29:54.801374 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:54.801379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:54.801447 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:29:54.801636 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:54.801640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:54.801716 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:54.801812 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:54.803034 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.803822 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.805036 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:54.805425 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:54.805521 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:54.809952 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:54.809966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:54.809983 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:29:54.810017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:54.812740 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.813427 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.813434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.813436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.813550 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:29:54.813869 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:29:54.813871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:29:54.813873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:29:54.814213 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:54.814421 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:54.814707 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:29:54.814855 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:29:54.816107 4959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:54.905676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4837
I20260620 18:29:54.922353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 4480
I20260620 18:29:54.923496 4541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:55.030463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 4480
2026-06-20T18:29:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4397 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:55Z Disabled control of system clock
I20260620 18:29:55.097059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36825
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:29:55.420562 5017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:55.420854 5017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:55.421026 5017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:55.428314 5017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:55.428426 5017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:55.428515 5017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:55.428606 5017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:55.428670 5017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:29:55.437888 5017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:55.439209 5017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:55.441012 5017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:55.453059 5022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:55.453118 5025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:55.453118 5023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:55.453750 5017 server_base.cc:1034] running on GCE node
I20260620 18:29:55.454644 5017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:55.456238 5017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:55.457515 5017 hybrid_clock.cc:648] HybridClock initialized: now 1781980195457479 us; error 37 us; skew 500 ppm
I20260620 18:29:55.457901 5017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:55.460255 5017 webserver.cc:460] Webserver started at http://127.0.97.62:39117/ using document root <none> and password file <none>
I20260620 18:29:55.460958 5017 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:55.461071 5017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:55.461410 5017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:55.464334 5017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "8ca3262c39a34da78cdbfe09bf0078b9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "579a4e5d74c4619dd30db0cfaf198a3d"
server_key_iv: "22378f2df6679cc00fcf5bc6bfaf3a93"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.465133 5017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ca3262c39a34da78cdbfe09bf0078b9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "579a4e5d74c4619dd30db0cfaf198a3d"
server_key_iv: "22378f2df6679cc00fcf5bc6bfaf3a93"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.470198 5017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:29:55.473603 5031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:55.474979 5017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:29:55.475158 5017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "8ca3262c39a34da78cdbfe09bf0078b9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "579a4e5d74c4619dd30db0cfaf198a3d"
server_key_iv: "22378f2df6679cc00fcf5bc6bfaf3a93"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.475373 5017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:55.481144 5017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:55.481895 5017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:55.482187 5017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:55.500722 5017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36825
I20260620 18:29:55.500757 5082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36825 every 8 connection(s)
I20260620 18:29:55.502075 5017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:55.505929 5083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:55.506613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5017
I20260620 18:29:55.506809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:55.507447 388 external_mini_cluster.cc:1428] Setting key 7db064775eee4bb7f9279ae58533a017
I20260620 18:29:55.515702 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:55.509535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.0.97.62:36825)
0620 18:29:55.510622 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:55.510633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:55.510680 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:29:55.512227 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:55.512249 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:55.512672 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:55.513003 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:55.514106 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:55.514112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:55.514124 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:55.514138 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:55.514197 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:55.514212 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:55.514367 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:55.514509 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:29:55.514766 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":443,"threads_started":1}
I20260620 18:29:55.519141 5083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: Bootstrap starting.
I20260620 18:29:55.521981 5083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:55.522926 5083 log.cc:824] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:55.525820 5083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: No bootstrap required, opened a new log
I20260620 18:29:55.533217 5083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } }
I20260620 18:29:55.533618 5083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:55.533725 5083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca3262c39a34da78cdbfe09bf0078b9, State: Initialized, Role: FOLLOWER
I20260620 18:29:55.534401 5083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } }
I20260620 18:29:55.534667 5083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:55.534819 5083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:55.535039 5083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:55.537441 5083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } }
I20260620 18:29:55.537897 5083 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca3262c39a34da78cdbfe09bf0078b9; no voters:
I20260620 18:29:55.538436 5083 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:55.538614 5088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:55.539362 5088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca3262c39a34da78cdbfe09bf0078b9, State: Running, Role: LEADER
I20260620 18:29:55.539984 5088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } }
I20260620 18:29:55.540675 5083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:55.542694 5090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca3262c39a34da78cdbfe09bf0078b9. Latest consensus state: current_term: 1 leader_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } } }
I20260620 18:29:55.543224 5090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:55.543171 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca3262c39a34da78cdbfe09bf0078b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36825 } } }
I20260620 18:29:55.543646 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:55.545974 5093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:55.550853 5093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:55.560664 5093 catalog_manager.cc:1347] Generated new cluster ID: bbcad304e07a4adeb060f4fca7c22ef3
I20260620 18:29:55.561035 5093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:55.572510 5093 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:55.572996 5093 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:55.585574 5093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: Generated new TSK 0
I20260620 18:29:55.586438 5093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:29:55.603092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--builtin_ntp_servers=127.0.97.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:29:55.930716 5107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:55.931016 5107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:55.931172 5107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:55.939155 5107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:55.939291 5107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:55.939483 5107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:29:55.949520 5107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:55.950891 5107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:55.952898 5107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:55.966243 5112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:55.966287 5115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:55.966549 5113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:55.967155 5107 server_base.cc:1034] running on GCE node
I20260620 18:29:55.967814 5107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:55.968734 5107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:55.969940 5107 hybrid_clock.cc:648] HybridClock initialized: now 1781980195969895 us; error 36 us; skew 500 ppm
I20260620 18:29:55.970219 5107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:55.972708 5107 webserver.cc:460] Webserver started at http://127.0.97.1:38383/ using document root <none> and password file <none>
I20260620 18:29:55.973425 5107 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:55.973512 5107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:55.973855 5107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:55.976780 5107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "4305bf5902b04cfe9a941c5c002b81d9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "21ef9ae00504c15fa83b833dba63c002"
server_key_iv: "29981480e3cefe3a1ac6ce8ed7032ec1"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.977605 5107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4305bf5902b04cfe9a941c5c002b81d9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "21ef9ae00504c15fa83b833dba63c002"
server_key_iv: "29981480e3cefe3a1ac6ce8ed7032ec1"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.983466 5107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:29:55.986992 5121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:55.988605 5107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:55.988799 5107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "4305bf5902b04cfe9a941c5c002b81d9"
format_stamp: "Formatted at 2026-06-20 18:29:55 on dist-test-slave-mcnm"
server_key: "21ef9ae00504c15fa83b833dba63c002"
server_key_iv: "29981480e3cefe3a1ac6ce8ed7032ec1"
server_key_version: "encryptionkey@0"
I20260620 18:29:55.989027 5107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:55.995965 5107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:55.996835 5107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:55.997123 5107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:55.998082 5107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:55.999624 5107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:55.999714 5107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.000075 5107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:56.000183 5107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.034597 5107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45077
I20260620 18:29:56.034634 5233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45077 every 8 connection(s)
I20260620 18:29:56.036001 5107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:29:56.044924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5107
I20260620 18:29:56.045025 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.038661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54733 (local address 127.0.97.62:36825)
0620 18:29:56.039127 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.039132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.040620 (+ 1488us) server_negotiation.cc:407] Connection header received
0620 18:29:56.042547 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.042551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.042605 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.042705 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.044090 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.044095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.044098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.044115 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.044160 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.044173 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.044293 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.044574 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.044649 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":243,"threads_started":1}
I20260620 18:29:56.045121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:29:56.045827 388 external_mini_cluster.cc:1428] Setting key 0bc5b0ca2f2eeb758211a9179049ea28
I20260620 18:29:56.045853 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.039297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36825 (local address 127.0.97.1:54733)
0620 18:29:56.040425 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.040471 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.042257 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.042915 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.042924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.043018 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.043772 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.043775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.043811 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.043896 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.044285 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.044297 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.044308 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.044575 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.044835 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":383,"threads_started":1}
I20260620 18:29:56.047593 5234 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36825
I20260620 18:29:56.047935 5234 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:56.048513 5234 heartbeater.cc:511] Master 127.0.97.62:36825 requested a full tablet report, sending...
I20260620 18:29:56.048899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--builtin_ntp_servers=127.0.97.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:29:56.050683 5047 ts_manager.cc:194] Registered new tserver with Master: 4305bf5902b04cfe9a941c5c002b81d9 (127.0.97.1:45077)
W20260620 18:29:56.372053 5238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:56.372371 5238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:56.372537 5238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:56.379741 5238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:56.379873 5238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:56.380096 5238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:29:56.389067 5238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:56.390365 5238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:56.392318 5238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:56.405580 5243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:56.405683 5244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:56.405706 5246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:56.406596 5238 server_base.cc:1034] running on GCE node
I20260620 18:29:56.407282 5238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:56.408342 5238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:56.409602 5238 hybrid_clock.cc:648] HybridClock initialized: now 1781980196409549 us; error 47 us; skew 500 ppm
I20260620 18:29:56.409977 5238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:56.412899 5238 webserver.cc:460] Webserver started at http://127.0.97.2:43489/ using document root <none> and password file <none>
I20260620 18:29:56.413705 5238 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:56.413866 5238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:56.414244 5238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:56.417562 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "842e0be4de9246a6a859e0f53d855f0b"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "dedfd57812ffc5b26e53ff5ce24ef7cd"
server_key_iv: "2570202218a8054760f08897ee867f33"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.418535 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "842e0be4de9246a6a859e0f53d855f0b"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "dedfd57812ffc5b26e53ff5ce24ef7cd"
server_key_iv: "2570202218a8054760f08897ee867f33"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.424471 5238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:29:56.428593 5252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.430105 5238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:29:56.430317 5238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "842e0be4de9246a6a859e0f53d855f0b"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "dedfd57812ffc5b26e53ff5ce24ef7cd"
server_key_iv: "2570202218a8054760f08897ee867f33"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.430572 5238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:56.436717 5238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:56.437537 5238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:56.437829 5238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:56.438807 5238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:56.440356 5238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:56.440485 5238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.440647 5238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:56.440745 5238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.476012 5238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42521
I20260620 18:29:56.476063 5364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42521 every 8 connection(s)
I20260620 18:29:56.477420 5238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:29:56.479408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5238
I20260620 18:29:56.479589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:29:56.480083 388 external_mini_cluster.cc:1428] Setting key f4f5ff5238d5ef984479d576c864dde7
I20260620 18:29:56.482962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--builtin_ntp_servers=127.0.97.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:29:56.486403 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.480481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56507 (local address 127.0.97.62:36825)
0620 18:29:56.480716 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.480723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.482342 (+ 1619us) server_negotiation.cc:407] Connection header received
0620 18:29:56.484156 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.484161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.484221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.484339 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.485618 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.485624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.485629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.485656 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.485696 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.485707 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.485824 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.486009 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.486070 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:29:56.487247 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.481124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36825 (local address 127.0.97.2:56507)
0620 18:29:56.482205 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.482246 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.483890 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.484556 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.484564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.484657 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.485335 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.485338 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.485374 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.485457 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.485800 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.485810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.485817 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.486088 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.486347 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260620 18:29:56.488677 5365 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36825
I20260620 18:29:56.489054 5365 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:56.489656 5365 heartbeater.cc:511] Master 127.0.97.62:36825 requested a full tablet report, sending...
I20260620 18:29:56.491160 5047 ts_manager.cc:194] Registered new tserver with Master: 842e0be4de9246a6a859e0f53d855f0b (127.0.97.2:42521)
W20260620 18:29:56.816192 5368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:56.816514 5368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:56.816674 5368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:56.823855 5368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:56.823983 5368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:56.824164 5368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:29:56.833132 5368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:56.834414 5368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:56.836294 5368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:56.849637 5373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:56.849620 5374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:56.850000 5376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:56.850464 5368 server_base.cc:1034] running on GCE node
I20260620 18:29:56.851054 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:56.852041 5368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:56.853247 5368 hybrid_clock.cc:648] HybridClock initialized: now 1781980196853161 us; error 77 us; skew 500 ppm
I20260620 18:29:56.853554 5368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:56.855715 5368 webserver.cc:460] Webserver started at http://127.0.97.3:42167/ using document root <none> and password file <none>
I20260620 18:29:56.856361 5368 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:56.856443 5368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:56.856740 5368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:56.859651 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "1319ec2e7179a7c71750e96c5dcb345c"
server_key_iv: "2cb618ab1c271c77e33525064b2f8d46"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.860397 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "1319ec2e7179a7c71750e96c5dcb345c"
server_key_iv: "2cb618ab1c271c77e33525064b2f8d46"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.865272 5368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:29:56.868501 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.869760 5368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:29:56.869889 5368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a"
format_stamp: "Formatted at 2026-06-20 18:29:56 on dist-test-slave-mcnm"
server_key: "1319ec2e7179a7c71750e96c5dcb345c"
server_key_iv: "2cb618ab1c271c77e33525064b2f8d46"
server_key_version: "encryptionkey@0"
I20260620 18:29:56.870061 5368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:56.875236 5368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:56.876015 5368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:56.876233 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:56.877048 5368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:29:56.878350 5368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:29:56.878432 5368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.878500 5368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:29:56.878556 5368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:56.912145 5368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44411
I20260620 18:29:56.912165 5494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44411 every 8 connection(s)
I20260620 18:29:56.913486 5368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:29:56.921933 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.916289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49285 (local address 127.0.97.62:36825)
0620 18:29:56.916491 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.916496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.917930 (+ 1434us) server_negotiation.cc:407] Connection header received
0620 18:29:56.919674 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.919680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.919754 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.919871 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.921137 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.921143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.921148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.921165 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.921205 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.921216 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.921346 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.921554 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.921655 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:29:56.922600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5368
I20260620 18:29:56.922768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:29:56.923141 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.916803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36825 (local address 127.0.97.3:49285)
0620 18:29:56.917758 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.917803 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.919348 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.920109 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.920118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.920211 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.920856 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.920859 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.920894 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.920977 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.921344 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.921354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.921365 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.921671 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.921922 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":304,"threads_started":1}
I20260620 18:29:56.923286 388 external_mini_cluster.cc:1428] Setting key 3933c6045b538ded3d7ac34677e11e76
I20260620 18:29:56.924680 5495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36825
I20260620 18:29:56.924988 5495 heartbeater.cc:464] Registering TS with master...
I20260620 18:29:56.925691 5495 heartbeater.cc:511] Master 127.0.97.62:36825 requested a full tablet report, sending...
I20260620 18:29:56.927104 5047 ts_manager.cc:194] Registered new tserver with Master: 2c1d27ffb9a845cdb1bcadd70bcbb10a (127.0.97.3:44411)
I20260620 18:29:56.938336 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:29:56.945642 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.943050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.0.97.62:36825)
0620 18:29:56.943286 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.943294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.943721 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:29:56.943939 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.943943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.943994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.944092 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.945022 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.945028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.945033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.945050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.945089 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.945102 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.945226 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.945313 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.945372 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:29:56.949540 5047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:29:56.956161 5047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:29:56.984474 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.968102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42521 (local address 127.0.0.1:35710)
0620 18:29:56.981319 (+ 13217us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.981336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.981429 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.982122 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.982129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.982224 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.983010 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.983016 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.983053 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.983139 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.983623 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.983634 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.983645 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.983817 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.983943 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13106,"thread_start_us":12923,"threads_started":1}
I20260620 18:29:56.985008 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.968378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.0.97.2:42521)
0620 18:29:56.981005 (+ 12627us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.981015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.981547 (+ 532us) server_negotiation.cc:407] Connection header received
0620 18:29:56.981620 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.981627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.981876 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.982085 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.983314 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.983327 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.983346 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.983366 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.983426 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.983452 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.984486 (+ 1034us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.984550 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.984649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12460,"thread_start_us":12356,"threads_started":1}
I20260620 18:29:56.985616 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.968138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.0.97.1:45077)
0620 18:29:56.982059 (+ 13921us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.982067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.982095 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:29:56.982154 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.982160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.982387 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.982545 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.984044 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.984053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.984064 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.984082 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.984133 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.984153 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.984297 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.985175 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.985292 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13746,"thread_start_us":13642,"threads_started":1}
I20260620 18:29:56.986231 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.967967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53562)
0620 18:29:56.980212 (+ 12245us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.980265 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.980444 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.982650 (+ 2206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.982655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.982676 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.983838 (+ 1162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.983842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.983857 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.983898 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.984684 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.984689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.984694 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.985830 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.985890 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11871,"thread_start_us":13065,"threads_started":1}
I20260620 18:29:56.987996 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.983547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44411 (local address 127.0.0.1:43174)
0620 18:29:56.984195 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:56.984211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:56.984313 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:56.985326 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:56.985330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:56.985350 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:56.986468 (+ 1118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:56.986471 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:56.986484 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:56.986520 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:56.986996 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:56.987001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:56.987006 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:29:56.987542 (+ 536us) client_negotiation.cc:240] Negotiation successful
0620 18:29:56.987595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":456,"threads_started":1}
I20260620 18:29:56.988718 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:56.982172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.0.97.3:44411)
0620 18:29:56.983916 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:56.983924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:56.984438 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:29:56.984517 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:56.984525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:56.984723 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:56.984864 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:56.986679 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:56.986687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:56.986698 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:56.986717 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:56.986766 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:56.986785 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:56.986943 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:56.988319 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0620 18:29:56.988418 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1573,"thread_start_us":1423,"threads_started":1}
I20260620 18:29:56.989333 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 (DEFAULT_TABLE table=test-table [id=b2e1e80f81364987a19e17269567c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:56.989691 5169 tablet_service.cc:1467] Processing CreateTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 (DEFAULT_TABLE table=test-table [id=b2e1e80f81364987a19e17269567c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:56.990806 5300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5876ce9de1d1486c9e5f20ced001eff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:56.990917 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5876ce9de1d1486c9e5f20ced001eff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:56.992046 5430 tablet_service.cc:1467] Processing CreateTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 (DEFAULT_TABLE table=test-table [id=b2e1e80f81364987a19e17269567c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:29:56.993218 5430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5876ce9de1d1486c9e5f20ced001eff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:57.004657 5513 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Bootstrap starting.
I20260620 18:29:57.008592 5513 tablet_bootstrap.cc:654] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:57.009656 5513 log.cc:824] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:57.012501 5513 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: No bootstrap required, opened a new log
I20260620 18:29:57.012797 5513 ts_tablet_manager.cc:1397] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:29:57.013492 5514 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Bootstrap starting.
I20260620 18:29:57.013602 5515 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Bootstrap starting.
I20260620 18:29:57.016410 5515 tablet_bootstrap.cc:654] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:57.016937 5514 tablet_bootstrap.cc:654] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:57.017438 5515 log.cc:824] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:57.018085 5514 log.cc:824] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:57.020140 5515 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: No bootstrap required, opened a new log
I20260620 18:29:57.020401 5515 ts_tablet_manager.cc:1397] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 18:29:57.021878 5514 tablet_bootstrap.cc:492] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: No bootstrap required, opened a new log
I20260620 18:29:57.022171 5514 ts_tablet_manager.cc:1397] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 18:29:57.023808 5513 raft_consensus.cc:348] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.024264 5513 raft_consensus.cc:374] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:57.024410 5513 raft_consensus.cc:729] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1d27ffb9a845cdb1bcadd70bcbb10a, State: Initialized, Role: FOLLOWER
I20260620 18:29:57.025115 5513 consensus_queue.cc:260] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.026227 5495 heartbeater.cc:503] Master 127.0.97.62:36825 was elected leader, sending a full tablet report...
I20260620 18:29:57.026885 5513 ts_tablet_manager.cc:1428] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 18:29:57.028650 5515 raft_consensus.cc:348] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.029078 5515 raft_consensus.cc:374] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:57.029173 5515 raft_consensus.cc:729] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4305bf5902b04cfe9a941c5c002b81d9, State: Initialized, Role: FOLLOWER
I20260620 18:29:57.029771 5515 consensus_queue.cc:260] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.030809 5234 heartbeater.cc:503] Master 127.0.97.62:36825 was elected leader, sending a full tablet report...
I20260620 18:29:57.031436 5515 ts_tablet_manager.cc:1428] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 18:29:57.033252 5514 raft_consensus.cc:348] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.033739 5514 raft_consensus.cc:374] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:57.033859 5514 raft_consensus.cc:729] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842e0be4de9246a6a859e0f53d855f0b, State: Initialized, Role: FOLLOWER
I20260620 18:29:57.034530 5514 consensus_queue.cc:260] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
W20260620 18:29:57.038302 5235 tablet.cc:2369] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:29:57.039086 5365 heartbeater.cc:503] Master 127.0.97.62:36825 was elected leader, sending a full tablet report...
I20260620 18:29:57.039789 5514 ts_tablet_manager.cc:1428] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260620 18:29:57.042016 5520 raft_consensus.cc:482] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:29:57.042322 5520 raft_consensus.cc:504] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.044282 5520 leader_election.cc:288] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842e0be4de9246a6a859e0f53d855f0b (127.0.97.2:42521), 2c1d27ffb9a845cdb1bcadd70bcbb10a (127.0.97.3:44411)
I20260620 18:29:57.053210 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.044681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53959 (local address 127.0.97.2:42521)
0620 18:29:57.044848 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.044852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.050917 (+ 6065us) server_negotiation.cc:407] Connection header received
0620 18:29:57.051176 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.051181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.051243 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.051349 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.052471 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.052480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.052487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.052512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.052558 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.052576 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.052735 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.052882 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.052946 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:29:57.053458 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.044521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42521 (local address 127.0.97.1:53959)
0620 18:29:57.050779 (+ 6258us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.050797 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:57.051011 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:57.051377 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:57.051383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:57.051408 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:57.052241 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:57.052247 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:57.052268 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:57.052316 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:57.052723 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:57.052727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:57.052731 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:29:57.052895 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:29:57.052945 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6112,"thread_start_us":5959,"threads_started":1}
I20260620 18:29:57.054210 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.051003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37499 (local address 127.0.97.3:44411)
0620 18:29:57.051229 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.051239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.051876 (+ 637us) server_negotiation.cc:407] Connection header received
0620 18:29:57.052015 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.052019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.052097 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.052273 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.053531 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.053538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.053544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.053567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.053605 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.053619 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.053736 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.053901 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.053960 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:29:57.054339 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.050866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44411 (local address 127.0.97.1:37499)
0620 18:29:57.051768 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.051782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:57.051911 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:57.052417 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:57.052421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:57.052446 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:29:57.053145 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:29:57.053147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:29:57.053159 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:29:57.053193 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:29:57.053785 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:29:57.053788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:29:57.053792 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:29:57.053938 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:29:57.053985 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":541,"threads_started":1}
I20260620 18:29:57.054400 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5876ce9de1d1486c9e5f20ced001eff2" candidate_uuid: "4305bf5902b04cfe9a941c5c002b81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842e0be4de9246a6a859e0f53d855f0b" is_pre_election: true
I20260620 18:29:57.054970 5320 raft_consensus.cc:2454] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4305bf5902b04cfe9a941c5c002b81d9 in term 0.
I20260620 18:29:57.055217 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5876ce9de1d1486c9e5f20ced001eff2" candidate_uuid: "4305bf5902b04cfe9a941c5c002b81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" is_pre_election: true
I20260620 18:29:57.055655 5124 leader_election.cc:302] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4305bf5902b04cfe9a941c5c002b81d9, 842e0be4de9246a6a859e0f53d855f0b; no voters:
I20260620 18:29:57.055771 5450 raft_consensus.cc:2454] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4305bf5902b04cfe9a941c5c002b81d9 in term 0.
I20260620 18:29:57.056077 5520 raft_consensus.cc:2781] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:29:57.056262 5520 raft_consensus.cc:482] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:29:57.056380 5520 raft_consensus.cc:3037] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:57.058463 5520 raft_consensus.cc:504] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.058988 5520 leader_election.cc:288] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [CANDIDATE]: Term 1 election: Requested vote from peers 842e0be4de9246a6a859e0f53d855f0b (127.0.97.2:42521), 2c1d27ffb9a845cdb1bcadd70bcbb10a (127.0.97.3:44411)
I20260620 18:29:57.059412 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5876ce9de1d1486c9e5f20ced001eff2" candidate_uuid: "4305bf5902b04cfe9a941c5c002b81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842e0be4de9246a6a859e0f53d855f0b"
I20260620 18:29:57.059473 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5876ce9de1d1486c9e5f20ced001eff2" candidate_uuid: "4305bf5902b04cfe9a941c5c002b81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a"
I20260620 18:29:57.059644 5320 raft_consensus.cc:3037] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:57.059783 5450 raft_consensus.cc:3037] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:57.062216 5320 raft_consensus.cc:2454] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4305bf5902b04cfe9a941c5c002b81d9 in term 1.
I20260620 18:29:57.062374 5450 raft_consensus.cc:2454] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4305bf5902b04cfe9a941c5c002b81d9 in term 1.
I20260620 18:29:57.062680 5124 leader_election.cc:302] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4305bf5902b04cfe9a941c5c002b81d9, 842e0be4de9246a6a859e0f53d855f0b; no voters:
I20260620 18:29:57.062968 5520 raft_consensus.cc:2781] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:57.063412 5520 raft_consensus.cc:686] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4305bf5902b04cfe9a941c5c002b81d9, State: Running, Role: LEADER
I20260620 18:29:57.063995 5520 consensus_queue.cc:237] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } }
I20260620 18:29:57.068434 5047 catalog_manager.cc:5495] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4305bf5902b04cfe9a941c5c002b81d9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4305bf5902b04cfe9a941c5c002b81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4305bf5902b04cfe9a941c5c002b81d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 } health_report { overall_health: UNKNOWN } } }
I20260620 18:29:57.090112 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.087836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.97.1:45077)
0620 18:29:57.088051 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.088060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.088113 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:29:57.088326 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.088330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.088407 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.088490 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.089439 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.089443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.089446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.089461 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.089493 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.089504 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.089601 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.089742 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.089788 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:29:57.098263 5320 raft_consensus.cc:1261] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 1 FOLLOWER]: Refusing update from remote peer 4305bf5902b04cfe9a941c5c002b81d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:57.098263 5450 raft_consensus.cc:1261] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 1 FOLLOWER]: Refusing update from remote peer 4305bf5902b04cfe9a941c5c002b81d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:29:57.099190 5524 consensus_queue.cc:1035] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842e0be4de9246a6a859e0f53d855f0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:57.099579 5520 consensus_queue.cc:1035] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c1d27ffb9a845cdb1bcadd70bcbb10a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:29:57.110945 5527 mvcc.cc:205] Tried to move back new op lower bound from 7298990887304814592 to 7298990887178203136. Current Snapshot: MvccSnapshot[applied={T|T < 7298990887304814592}]
I20260620 18:29:57.124531 5048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:57.124791 5048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:29:57.128264 5048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ca3262c39a34da78cdbfe09bf0078b9: Sending DeleteTablet for 3 replicas of tablet 5876ce9de1d1486c9e5f20ced001eff2
I20260620 18:29:57.129323 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:57 UTC) from {username='slave'} at 127.0.0.1:53562
I20260620 18:29:57.129689 5430 tablet_service.cc:1514] Processing DeleteTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:57 UTC) from {username='slave'} at 127.0.0.1:43174
I20260620 18:29:57.130503 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet 5876ce9de1d1486c9e5f20ced001eff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:29:57 UTC) from {username='slave'} at 127.0.0.1:35710
I20260620 18:29:57.131608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5107
I20260620 18:29:57.132699 5538 tablet_replica.cc:330] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: stopping tablet replica
I20260620 18:29:57.133219 5538 raft_consensus.cc:2229] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:57.133775 5538 raft_consensus.cc:2258] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:57.134567 5539 tablet_replica.cc:330] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: stopping tablet replica
I20260620 18:29:57.134892 5537 tablet_replica.cc:330] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: stopping tablet replica
I20260620 18:29:57.134943 5539 raft_consensus.cc:2229] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:29:57.135239 5537 raft_consensus.cc:2229] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:29:57.135385 5539 raft_consensus.cc:2258] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:57.136492 5537 raft_consensus.cc:2258] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:29:57.137290 5538 ts_tablet_manager.cc:1905] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:57.137883 5539 ts_tablet_manager.cc:1905] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:57.144001 5538 ts_tablet_manager.cc:1918] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:57.144170 5538 log.cc:1196] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/5876ce9de1d1486c9e5f20ced001eff2
I20260620 18:29:57.144668 5538 ts_tablet_manager.cc:1939] T 5876ce9de1d1486c9e5f20ced001eff2 P 2c1d27ffb9a845cdb1bcadd70bcbb10a: Deleting consensus metadata
I20260620 18:29:57.145248 5537 ts_tablet_manager.cc:1905] T 5876ce9de1d1486c9e5f20ced001eff2 P 4305bf5902b04cfe9a941c5c002b81d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:29:57.146770 5034 catalog_manager.cc:4841] TS 2c1d27ffb9a845cdb1bcadd70bcbb10a (127.0.97.3:44411): tablet 5876ce9de1d1486c9e5f20ced001eff2 (table test-table [id=b2e1e80f81364987a19e17269567c4d8]) successfully deleted
I20260620 18:29:57.147315 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.132758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.97.1:45077)
0620 18:29:57.132937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.132943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.144493 (+ 11550us) server_negotiation.cc:407] Connection header received
0620 18:29:57.144935 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.144940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.145008 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.145096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.146396 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.146401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.146410 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.146429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.146464 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.146479 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.146579 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.146879 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.146943 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:29:57.148521 5229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:57.150992 5539 ts_tablet_manager.cc:1918] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:29:57.151202 5539 log.cc:1196] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/5876ce9de1d1486c9e5f20ced001eff2
I20260620 18:29:57.151866 5539 ts_tablet_manager.cc:1939] T 5876ce9de1d1486c9e5f20ced001eff2 P 842e0be4de9246a6a859e0f53d855f0b: Deleting consensus metadata
I20260620 18:29:57.153479 5034 catalog_manager.cc:4841] TS 842e0be4de9246a6a859e0f53d855f0b (127.0.97.2:42521): tablet 5876ce9de1d1486c9e5f20ced001eff2 (table test-table [id=b2e1e80f81364987a19e17269567c4d8]) successfully deleted
I20260620 18:29:57.242359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5107
W20260620 18:29:57.259389 5035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45077 (error 108)
I20260620 18:29:57.260068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5238
I20260620 18:29:57.261543 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.260809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53586)
0620 18:29:57.261255 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.261367 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
W20260620 18:29:57.261775 5035 catalog_manager.cc:4584] TS 4305bf5902b04cfe9a941c5c002b81d9 (127.0.97.1:45077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5876ce9de1d1486c9e5f20ced001eff2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
I20260620 18:29:57.263031 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.260861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.0.97.2:42521)
0620 18:29:57.261056 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.261062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.261082 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:29:57.261213 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.261218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.261336 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.261419 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.262356 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.262361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.262365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.262388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.262415 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.262426 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.262557 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.262686 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.262740 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:29:57.263792 5360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:57.315706 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.315059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53590)
0620 18:29:57.315334 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.315456 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:29:57.316826 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.316435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53592)
0620 18:29:57.316597 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.316699 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:29:57.350401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5238
I20260620 18:29:57.366616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5368
I20260620 18:29:57.369252 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.367401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.0.97.3:44411)
0620 18:29:57.367621 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:57.367625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:57.367642 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:29:57.367698 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:57.367701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:57.367783 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:57.367869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:57.368756 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:57.368761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:57.368764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:57.368778 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:57.368808 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:57.368819 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:57.368919 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:57.368988 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:29:57.369027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:29:57.369872 5490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:57.377265 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.376749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53602)
0620 18:29:57.376962 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.377075 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:29:57.378381 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.377958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53608)
0620 18:29:57.378134 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.378230 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:29:57.457036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5368
I20260620 18:29:57.459626 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.459009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53610)
0620 18:29:57.459286 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.459395 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:29:57.460695 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:57.460275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45077 (local address 127.0.0.1:53620)
0620 18:29:57.460506 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:57.460582 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:29:57.473489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5017
I20260620 18:29:57.474653 5078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:57.577509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5017
2026-06-20T18:29:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2545 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:29:57.618697 388 mini_oidc.cc:152] Starting JWKS server
I20260620 18:29:57.621292 388 webserver.cc:460] Webserver started at https://127.0.0.1:36715/ using document root <none> and password file <none>
I20260620 18:29:57.621635 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:29:57.622339 388 webserver.cc:460] Webserver started at http://127.0.0.1:37093/ using document root <none> and password file <none>
2026-06-20T18:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:57Z Disabled control of system clock
I20260620 18:29:57.650578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46291
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46291
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/testchainca.cert with env {}
W20260620 18:29:57.964701 5551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:57.965051 5551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:57.965226 5551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:57.972270 5551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:57.972420 5551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:57.972515 5551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:57.972623 5551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:57.972718 5551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:29:57.972806 5551 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:29:57.972880 5551 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36715/jwks.json
I20260620 18:29:57.982429 5551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:57.983709 5551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:57.984989 5551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:57.996658 5557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:57.996704 5556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:57.996672 5559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:57.997766 5551 server_base.cc:1034] running on GCE node
I20260620 18:29:57.998490 5551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:57.999984 5551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:58.001210 5551 hybrid_clock.cc:648] HybridClock initialized: now 1781980198001162 us; error 41 us; skew 500 ppm
I20260620 18:29:58.001607 5551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:58.004069 5551 webserver.cc:460] Webserver started at http://127.0.97.62:42041/ using document root <none> and password file <none>
I20260620 18:29:58.004772 5551 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:58.004863 5551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:58.005188 5551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:58.008034 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "52e5df8f9c744f718605f098041aa13c"
format_stamp: "Formatted at 2026-06-20 18:29:57 on dist-test-slave-mcnm"
server_key: "b8ec306c9c2ea8a21f2e3c5bba0381f4"
server_key_iv: "ad54140088e7756f92dd9b3a9de0de13"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.008865 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "52e5df8f9c744f718605f098041aa13c"
format_stamp: "Formatted at 2026-06-20 18:29:57 on dist-test-slave-mcnm"
server_key: "b8ec306c9c2ea8a21f2e3c5bba0381f4"
server_key_iv: "ad54140088e7756f92dd9b3a9de0de13"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.013926 5551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:29:58.017448 5565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:58.018790 5551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:29:58.018970 5551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "52e5df8f9c744f718605f098041aa13c"
format_stamp: "Formatted at 2026-06-20 18:29:57 on dist-test-slave-mcnm"
server_key: "b8ec306c9c2ea8a21f2e3c5bba0381f4"
server_key_iv: "ad54140088e7756f92dd9b3a9de0de13"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.019183 5551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:58.032917 5551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:58.033777 5551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:58.034080 5551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:58.052734 5551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46291
I20260620 18:29:58.052755 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46291 every 8 connection(s)
I20260620 18:29:58.054073 5551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:58.057982 5617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:58.059247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5551
I20260620 18:29:58.059506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:58.060448 388 external_mini_cluster.cc:1428] Setting key 92c61a46b6048288350416719029abde
I20260620 18:29:58.071960 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c: Bootstrap starting.
I20260620 18:29:58.073880 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:58.062486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.0.97.62:46291)
0620 18:29:58.063705 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:58.063717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:58.063771 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:29:58.065430 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:58.065452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:58.065911 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:58.066330 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:58.066876 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.068187 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.069541 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.070187 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.070380 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.071657 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:58.071666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:58.071678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:58.072267 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:58.072343 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:58.072363 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:58.072485 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:58.072697 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 18:29:58.072980 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":426,"threads_started":1}
I20260620 18:29:58.075332 5617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:58.076592 5617 log.cc:824] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:58.077433 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:29:58.079342 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c: No bootstrap required, opened a new log
I20260620 18:29:58.086834 5617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } }
I20260620 18:29:58.087203 5617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:58.087319 5617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e5df8f9c744f718605f098041aa13c, State: Initialized, Role: FOLLOWER
I20260620 18:29:58.088047 5617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } }
I20260620 18:29:58.088287 5617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:58.088426 5617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:58.088611 5617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:58.091048 5617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } }
I20260620 18:29:58.091624 5617 leader_election.cc:302] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e5df8f9c744f718605f098041aa13c; no voters:
I20260620 18:29:58.092207 5617 leader_election.cc:288] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:58.092378 5622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:58.093124 5622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [term 1 LEADER]: Becoming Leader. State: Replica: 52e5df8f9c744f718605f098041aa13c, State: Running, Role: LEADER
I20260620 18:29:58.093664 5622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } }
I20260620 18:29:58.094825 5617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:58.097043 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e5df8f9c744f718605f098041aa13c. Latest consensus state: current_term: 1 leader_uuid: "52e5df8f9c744f718605f098041aa13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } } }
I20260620 18:29:58.097868 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:58.098680 5623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e5df8f9c744f718605f098041aa13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5df8f9c744f718605f098041aa13c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46291 } } }
I20260620 18:29:58.099036 5623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:58.101589 5629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:58.107491 5629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:58.116034 5629 catalog_manager.cc:1347] Generated new cluster ID: f3137d86ab46467a9482c2f7d5b1c780
I20260620 18:29:58.116151 5629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:58.131711 5629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:58.133824 5629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:58.147586 5629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e5df8f9c744f718605f098041aa13c: Generated new TSK 0
I20260620 18:29:58.148669 5629 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:29:58.198428 5647 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:29:58.198621 5647 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:29:58.199762 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:58.174629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.0.97.62:46291)
0620 18:29:58.174831 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:58.174837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:58.175478 (+ 641us) server_negotiation.cc:407] Connection header received
0620 18:29:58.175837 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:58.175843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:58.175941 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:58.176095 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:29:58.176677 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.177819 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.179405 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.180158 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.180292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.180836 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:29:58.199033 (+ 18197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:29:58.199395 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:29:58.199880 5620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:29:58.201345 5646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:29:58.174629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46291 (local address 127.0.0.1:58470)
0620 18:29:58.175376 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:58.175394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:58.175611 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:58.176143 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:58.176148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:58.176160 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:29:58.176506 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:58.176514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.177983 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.177987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:58.179260 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:58.179273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.180298 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.180357 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:58.180489 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.180544 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:29:58.199346 (+ 18802us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:29:58.199752 (+ 406us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":418,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:46291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:29:58.203279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5551
I20260620 18:29:58.204609 5612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:58.297083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5551
2026-06-20T18:29:58Z chronyd exiting
I20260620 18:29:58.347187 388 test_util.cc:175] -----------------------------------------------
I20260620 18:29:58.347363 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980152255989-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:29:58.350981 388 mini_oidc.cc:152] Starting JWKS server
I20260620 18:29:58.352851 388 webserver.cc:460] Webserver started at https://127.0.0.1:34487/ using document root <none> and password file <none>
I20260620 18:29:58.353096 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:29:58.353789 388 webserver.cc:460] Webserver started at http://127.0.0.1:43807/ using document root <none> and password file <none>
2026-06-20T18:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:58Z Disabled control of system clock
I20260620 18:29:58.379359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33981
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33981
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34487/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/testchainca.cert with env {}
W20260620 18:29:58.687686 5654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:58.687968 5654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:58.688055 5654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:58.695067 5654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:58.695175 5654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:58.695214 5654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:58.695261 5654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:58.695304 5654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:29:58.695356 5654 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:29:58.695399 5654 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34487/jwks.json
I20260620 18:29:58.705358 5654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34487/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:58.706593 5654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:58.707834 5654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:58.719329 5659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:58.719408 5660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:58.719447 5662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:58.720981 5654 server_base.cc:1034] running on GCE node
I20260620 18:29:58.721738 5654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:58.723143 5654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:58.724360 5654 hybrid_clock.cc:648] HybridClock initialized: now 1781980198724309 us; error 52 us; skew 500 ppm
I20260620 18:29:58.724704 5654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:58.732211 5654 webserver.cc:460] Webserver started at http://127.0.97.62:34351/ using document root <none> and password file <none>
I20260620 18:29:58.732865 5654 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:58.732955 5654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:58.733224 5654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:58.736167 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "ff9299d906b04f50b557bc130f0f1a3e"
format_stamp: "Formatted at 2026-06-20 18:29:58 on dist-test-slave-mcnm"
server_key: "8e08c548fa818e71b25913abbd83a671"
server_key_iv: "032beba8996fd1dbf1ab0b0b4f591454"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.736889 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff9299d906b04f50b557bc130f0f1a3e"
format_stamp: "Formatted at 2026-06-20 18:29:58 on dist-test-slave-mcnm"
server_key: "8e08c548fa818e71b25913abbd83a671"
server_key_iv: "032beba8996fd1dbf1ab0b0b4f591454"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.741906 5654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:29:58.745317 5668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:58.746732 5654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:29:58.746929 5654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "ff9299d906b04f50b557bc130f0f1a3e"
format_stamp: "Formatted at 2026-06-20 18:29:58 on dist-test-slave-mcnm"
server_key: "8e08c548fa818e71b25913abbd83a671"
server_key_iv: "032beba8996fd1dbf1ab0b0b4f591454"
server_key_version: "encryptionkey@0"
I20260620 18:29:58.747124 5654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:58.765636 5654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:58.766623 5654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:58.766913 5654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:58.786305 5654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33981
I20260620 18:29:58.786336 5719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33981 every 8 connection(s)
I20260620 18:29:58.787616 5654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:58.791311 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:58.796274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5654
I20260620 18:29:58.796473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:58.797231 388 external_mini_cluster.cc:1428] Setting key a422ef62d0aba45b9873398197a98c5b
I20260620 18:29:58.802618 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e: Bootstrap starting.
I20260620 18:29:58.805706 5720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:58.806721 5720 log.cc:824] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:58.809644 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e: No bootstrap required, opened a new log
I20260620 18:29:58.810683 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:58.799196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.97.62:33981)
0620 18:29:58.800278 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:58.800289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:58.800333 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:29:58.802085 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:58.802109 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:58.802555 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:58.802966 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:58.803575 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.804901 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.806242 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.806974 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.807161 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.808408 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:58.808428 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:58.808442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:58.808956 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:58.809038 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:58.809057 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:58.809187 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:58.809410 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:29:58.809716 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":392,"threads_started":1}
I20260620 18:29:58.814456 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:29:58.817364 5720 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } }
I20260620 18:29:58.817768 5720 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:58.817878 5720 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9299d906b04f50b557bc130f0f1a3e, State: Initialized, Role: FOLLOWER
I20260620 18:29:58.818352 5720 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } }
I20260620 18:29:58.818536 5720 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:58.818632 5720 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:58.818751 5720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:58.821317 5720 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } }
I20260620 18:29:58.821877 5720 leader_election.cc:302] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff9299d906b04f50b557bc130f0f1a3e; no voters:
I20260620 18:29:58.822489 5720 leader_election.cc:288] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:58.822651 5725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:58.823380 5725 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [term 1 LEADER]: Becoming Leader. State: Replica: ff9299d906b04f50b557bc130f0f1a3e, State: Running, Role: LEADER
I20260620 18:29:58.823921 5725 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } }
I20260620 18:29:58.824584 5720 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:58.826500 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff9299d906b04f50b557bc130f0f1a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } } }
I20260620 18:29:58.826578 5727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9299d906b04f50b557bc130f0f1a3e. Latest consensus state: current_term: 1 leader_uuid: "ff9299d906b04f50b557bc130f0f1a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9299d906b04f50b557bc130f0f1a3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33981 } } }
I20260620 18:29:58.826934 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:58.827082 5727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:58.828464 5730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:58.833528 5730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:58.842051 5730 catalog_manager.cc:1347] Generated new cluster ID: 3125a70448564bc0be1b48cf1642e49d
I20260620 18:29:58.842155 5730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:58.858486 5730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:58.860023 5730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:58.867089 5730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff9299d906b04f50b557bc130f0f1a3e: Generated new TSK 0
I20260620 18:29:58.867966 5730 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:29:58.909633 5750 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:29:58.909781 5750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:29:58.910957 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:58.890740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.0.97.62:33981)
0620 18:29:58.890927 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:58.890933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:58.891295 (+ 362us) server_negotiation.cc:407] Connection header received
0620 18:29:58.891480 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:58.891485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:58.891594 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:58.891765 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:29:58.892244 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.893387 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.894780 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.895478 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.895637 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.896087 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:29:58.910300 (+ 14213us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34487/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:29:58.910681 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34487/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:29:58.911085 5723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34487/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 18:29:58.911095 5749 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:29:58.890587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33981 (local address 127.0.0.1:40280)
0620 18:29:58.891186 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:58.891203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:58.891320 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:58.891798 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:58.891803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:58.891813 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:29:58.892064 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:58.892072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.893548 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.893553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:58.894638 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:58.894650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:58.895629 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:58.895669 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:58.895809 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:58.895841 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:29:58.910563 (+ 14722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34487/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:29:58.910771 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33981: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34487/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":330,"threads_started":1}
I20260620 18:29:58.911861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5654
I20260620 18:29:58.913204 5715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:29:59.004520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5654
2026-06-20T18:29:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:29:59.061515 388 mini_oidc.cc:152] Starting JWKS server
I20260620 18:29:59.063367 388 webserver.cc:460] Webserver started at https://127.0.0.1:35381/ using document root <none> and password file <none>
I20260620 18:29:59.063661 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:29:59.064286 388 webserver.cc:460] Webserver started at http://127.0.0.1:35649/ using document root <none> and password file <none>
2026-06-20T18:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:59Z Disabled control of system clock
I20260620 18:29:59.089787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36113
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36113
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:29:59.402837 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:29:59.403113 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:29:59.403198 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:29:59.410393 5757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:29:59.410497 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:29:59.410535 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:29:59.410584 5757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:29:59.410630 5757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:29:59.410687 5757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:29:59.410727 5757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35381/jwks.json
I20260620 18:29:59.419811 5757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:29:59.421025 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:29:59.422673 5757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:29:59.434311 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:59.434415 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:29:59.434391 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:29:59.435076 5757 server_base.cc:1034] running on GCE node
I20260620 18:29:59.435868 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:29:59.437253 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:29:59.438493 5757 hybrid_clock.cc:648] HybridClock initialized: now 1781980199438469 us; error 52 us; skew 500 ppm
I20260620 18:29:59.438851 5757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:29:59.441401 5757 webserver.cc:460] Webserver started at http://127.0.97.62:43739/ using document root <none> and password file <none>
I20260620 18:29:59.442086 5757 fs_manager.cc:359] Metadata directory not provided
I20260620 18:29:59.442215 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:29:59.442559 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:29:59.445542 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "2d7f300a66944c4ea2ebf9962b80d975"
format_stamp: "Formatted at 2026-06-20 18:29:59 on dist-test-slave-mcnm"
server_key: "83d93d297137e6ad6412c5300c75bade"
server_key_iv: "f17f7b4ed7df790031b1b9e9030182f8"
server_key_version: "encryptionkey@0"
I20260620 18:29:59.446324 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d7f300a66944c4ea2ebf9962b80d975"
format_stamp: "Formatted at 2026-06-20 18:29:59 on dist-test-slave-mcnm"
server_key: "83d93d297137e6ad6412c5300c75bade"
server_key_iv: "f17f7b4ed7df790031b1b9e9030182f8"
server_key_version: "encryptionkey@0"
I20260620 18:29:59.451362 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:29:59.454756 5771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:29:59.456146 5757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:29:59.456327 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "2d7f300a66944c4ea2ebf9962b80d975"
format_stamp: "Formatted at 2026-06-20 18:29:59 on dist-test-slave-mcnm"
server_key: "83d93d297137e6ad6412c5300c75bade"
server_key_iv: "f17f7b4ed7df790031b1b9e9030182f8"
server_key_version: "encryptionkey@0"
I20260620 18:29:59.456547 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:29:59.469372 5757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:29:59.470139 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:29:59.470415 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:29:59.488336 5757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36113
I20260620 18:29:59.488355 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36113 every 8 connection(s)
I20260620 18:29:59.489620 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:29:59.493232 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:29:59.496604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5757
I20260620 18:29:59.496829 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:29:59.497550 388 external_mini_cluster.cc:1428] Setting key a9f317035b1dcc874e38ef1a265f90f4
I20260620 18:29:59.507277 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975: Bootstrap starting.
I20260620 18:29:59.511147 5823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975: Neither blocks nor log segments found. Creating new log.
I20260620 18:29:59.512571 5823 log.cc:824] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975: Log is configured to *not* fsync() on all Append() calls
I20260620 18:29:59.513597 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:59.499620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.0.97.62:36113)
0620 18:29:59.500625 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:59.500640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:59.500693 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:29:59.502747 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:59.502783 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:59.503409 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:59.503927 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:29:59.504497 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.506446 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.508207 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.509329 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.509540 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:59.510605 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:29:59.510615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:29:59.510633 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:29:59.511401 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:29:59.511562 (+ 161us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:29:59.511602 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:29:59.511741 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:29:59.511960 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 18:29:59.512290 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":290,"threads_started":1}
I20260620 18:29:59.515962 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975: No bootstrap required, opened a new log
I20260620 18:29:59.517973 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:29:59.524153 5823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } }
I20260620 18:29:59.524533 5823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:29:59.524657 5823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7f300a66944c4ea2ebf9962b80d975, State: Initialized, Role: FOLLOWER
I20260620 18:29:59.525153 5823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } }
I20260620 18:29:59.525346 5823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:29:59.525429 5823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:29:59.525553 5823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:29:59.528088 5823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } }
I20260620 18:29:59.528528 5823 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7f300a66944c4ea2ebf9962b80d975; no voters:
I20260620 18:29:59.529037 5823 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:29:59.529197 5828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:29:59.529981 5828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7f300a66944c4ea2ebf9962b80d975, State: Running, Role: LEADER
I20260620 18:29:59.530560 5828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } }
I20260620 18:29:59.531157 5823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:29:59.533620 5829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7f300a66944c4ea2ebf9962b80d975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } } }
I20260620 18:29:59.534051 5829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:59.534144 5830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7f300a66944c4ea2ebf9962b80d975. Latest consensus state: current_term: 1 leader_uuid: "2d7f300a66944c4ea2ebf9962b80d975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7f300a66944c4ea2ebf9962b80d975" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36113 } } }
I20260620 18:29:59.534476 5830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975 [sys.catalog]: This master's current role is: LEADER
I20260620 18:29:59.535617 5837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:29:59.540730 5837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:29:59.549568 5837 catalog_manager.cc:1347] Generated new cluster ID: 77c332a6e9fe4803b1a9075658529c1e
I20260620 18:29:59.549712 5837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:29:59.563290 5837 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:29:59.564592 5837 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:29:59.574153 5837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d7f300a66944c4ea2ebf9962b80d975: Generated new TSK 0
I20260620 18:29:59.575001 5837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:29:59.603802 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:29:59.595410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.0.97.62:36113)
0620 18:29:59.595672 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:29:59.595682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:29:59.596172 (+ 490us) server_negotiation.cc:407] Connection header received
0620 18:29:59.596401 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:29:59.596406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:29:59.596497 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:29:59.596621 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:29:59.597164 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.597863 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.599948 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.600416 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.600519 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:59.601124 (+ 605us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:29:59.603187 (+ 2063us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:29:59.603516 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:29:59.603960 5826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:29:59.606881 5852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:29:59.595377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36113 (local address 127.0.0.1:49982)
0620 18:29:59.596046 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:29:59.596067 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:29:59.596222 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:29:59.596673 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:29:59.596678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:29:59.596692 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:29:59.597008 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:59.597019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.598013 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.598019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:59.599780 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:29:59.599797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:29:59.600559 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:29:59.600634 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:29:59.600822 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:29:59.600865 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:29:59.603386 (+ 2521us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:29:59.606579 (+ 3193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":2899,"thread_start_us":354,"threads_started":1}
I20260620 18:29:59.607935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5757
I20260620 18:29:59.609225 5818 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:29:59.617507 5853 webserver.cc:533] Webserver: sslize failed:
E20260620 18:29:59.617631 5853 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:29:59.699636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5757
2026-06-20T18:29:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:29:59Z Disabled control of system clock
I20260620 18:29:59.780179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37257
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37257
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/testchainca.cert with env {}
W20260620 18:30:00.099962 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:00.100248 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:00.100333 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:00.107582 5858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:00.107681 5858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:00.107717 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:00.107764 5858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:00.107806 5858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:30:00.107865 5858 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:30:00.107905 5858 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:30:00.117627 5858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:00.118855 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:00.120092 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:00.131585 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:00.131682 5866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:00.131649 5864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:00.133003 5858 server_base.cc:1034] running on GCE node
I20260620 18:30:00.133793 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:00.135167 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:00.136407 5858 hybrid_clock.cc:648] HybridClock initialized: now 1781980200136407 us; error 125 us; skew 500 ppm
I20260620 18:30:00.136770 5858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:00.139238 5858 webserver.cc:460] Webserver started at http://127.0.97.62:36235/ using document root <none> and password file <none>
I20260620 18:30:00.139992 5858 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:00.140128 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:00.140455 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:00.143232 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "0f6c6295163c42a692c22af71c486a57"
format_stamp: "Formatted at 2026-06-20 18:30:00 on dist-test-slave-mcnm"
server_key: "79abaebbc9939cd400dcbc6e54049a46"
server_key_iv: "8b187c6561f1069a71fa674c8efe3288"
server_key_version: "encryptionkey@0"
I20260620 18:30:00.144073 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f6c6295163c42a692c22af71c486a57"
format_stamp: "Formatted at 2026-06-20 18:30:00 on dist-test-slave-mcnm"
server_key: "79abaebbc9939cd400dcbc6e54049a46"
server_key_iv: "8b187c6561f1069a71fa674c8efe3288"
server_key_version: "encryptionkey@0"
I20260620 18:30:00.148937 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:30:00.152159 5872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:00.153411 5858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:30:00.153606 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "0f6c6295163c42a692c22af71c486a57"
format_stamp: "Formatted at 2026-06-20 18:30:00 on dist-test-slave-mcnm"
server_key: "79abaebbc9939cd400dcbc6e54049a46"
server_key_iv: "8b187c6561f1069a71fa674c8efe3288"
server_key_version: "encryptionkey@0"
I20260620 18:30:00.153811 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:00.174372 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:00.175182 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:00.175465 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:00.194144 5858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37257
I20260620 18:30:00.194160 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37257 every 8 connection(s)
I20260620 18:30:00.196029 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:00.197966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5858
I20260620 18:30:00.198181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:00.199090 388 external_mini_cluster.cc:1428] Setting key 53818491e3b9b6fe2af696447e2eb06c
I20260620 18:30:00.200744 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:00.214942 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57: Bootstrap starting.
I20260620 18:30:00.216490 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:00.201135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.0.97.62:37257)
0620 18:30:00.202434 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:00.202455 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:00.204137 (+ 1682us) server_negotiation.cc:407] Connection header received
0620 18:30:00.207189 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:00.207211 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:00.207757 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:00.208086 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:00.208739 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.209955 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.211921 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.212645 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.212771 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:00.214134 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:00.214141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:00.214154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:00.214676 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:00.214752 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:00.214772 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:00.214855 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:00.215156 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:30:00.215497 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":382,"threads_started":1}
I20260620 18:30:00.218621 5924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:00.219938 5924 log.cc:824] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:00.220543 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:30:00.222783 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57: No bootstrap required, opened a new log
I20260620 18:30:00.229733 5924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } }
I20260620 18:30:00.230036 5924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:00.230145 5924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6c6295163c42a692c22af71c486a57, State: Initialized, Role: FOLLOWER
I20260620 18:30:00.230805 5924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } }
I20260620 18:30:00.231057 5924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:00.231189 5924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:00.231369 5924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:00.233394 5924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } }
I20260620 18:30:00.233867 5924 leader_election.cc:302] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f6c6295163c42a692c22af71c486a57; no voters:
I20260620 18:30:00.234489 5924 leader_election.cc:288] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:00.234620 5929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:00.235312 5929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6c6295163c42a692c22af71c486a57, State: Running, Role: LEADER
I20260620 18:30:00.235831 5929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } }
I20260620 18:30:00.236482 5924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:00.238277 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f6c6295163c42a692c22af71c486a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } } }
I20260620 18:30:00.238607 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:00.243242 5935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:00.243376 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f6c6295163c42a692c22af71c486a57. Latest consensus state: current_term: 1 leader_uuid: "0f6c6295163c42a692c22af71c486a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c6295163c42a692c22af71c486a57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37257 } } }
I20260620 18:30:00.243815 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:00.248551 5935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:00.256605 5935 catalog_manager.cc:1347] Generated new cluster ID: 3a4b578fb3cb4284bdd2e10ec687c76f
I20260620 18:30:00.256721 5935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:00.266083 5935 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:00.267324 5935 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:00.275840 5935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f6c6295163c42a692c22af71c486a57: Generated new TSK 0
I20260620 18:30:00.276795 5935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:30:00.385169 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:00.293141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.97.62:37257)
0620 18:30:00.293332 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:00.293339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:00.293764 (+ 425us) server_negotiation.cc:407] Connection header received
0620 18:30:00.294004 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:00.294010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:00.294112 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:00.294281 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:30:00.294770 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.295912 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.297316 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.297972 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.298107 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:00.298572 (+ 465us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:30:00.384436 (+ 85864us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:30:00.384840 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37308: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:30:00.385314 5927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37308: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:30:00.385401 5954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:00.293022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37257 (local address 127.0.0.1:37308)
0620 18:30:00.293657 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:00.293673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:00.293788 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:00.294336 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:00.294340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:00.294350 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:30:00.294602 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:00.294610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.296079 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.296084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:00.297168 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:00.297180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:00.298105 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:00.298150 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:00.298270 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:00.298301 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:30:00.384739 (+ 86438us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:30:00.385029 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37257: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":330,"threads_started":1}
I20260620 18:30:00.387032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5858
I20260620 18:30:00.388319 5919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:00.487249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5858
2026-06-20T18:30:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (757 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (411 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1468 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:02 dist-test-slave-mcnm krb5kdc[5970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:02 dist-test-slave-mcnm krb5kdc[5970](info): set up 2 sockets
Jun 20 18:30:02 dist-test-slave-mcnm krb5kdc[5970](info): commencing operation
krb5kdc: starting...
W20260620 18:30:03.472878 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:03 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:30:03.677248 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:03.995111 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:03.995399 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:03.995575 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:04.002735 5983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:04.002842 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:04.002945 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:04.003041 5983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:04.003108 5983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:04.013330 5983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:04.014596 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:04.015903 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:04.027667 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:04.027710 5991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:04.027750 5989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:04.028825 5983 server_base.cc:1034] running on GCE node
I20260620 18:30:04.029580 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:04.030953 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:04.032176 5983 hybrid_clock.cc:648] HybridClock initialized: now 1781980204032128 us; error 33 us; skew 500 ppm
I20260620 18:30:04.032552 5983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:04 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:04.040406 5983 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:30:04.041929 5983 webserver.cc:460] Webserver started at http://127.0.97.62:40509/ using document root <none> and password file <none>
I20260620 18:30:04.042625 5983 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:04.042769 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:04.043084 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:04.046026 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "5337b1264e014d9cb5052c907677c0f6"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "e2d8f084fd911664854bc0580be51489"
server_key_iv: "ccca67b72d37d8c946c9f9bc12be61c8"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.046806 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "5337b1264e014d9cb5052c907677c0f6"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "e2d8f084fd911664854bc0580be51489"
server_key_iv: "ccca67b72d37d8c946c9f9bc12be61c8"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.051891 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:30:04.055219 5998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:04.056529 5983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:04.056751 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "5337b1264e014d9cb5052c907677c0f6"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "e2d8f084fd911664854bc0580be51489"
server_key_iv: "ccca67b72d37d8c946c9f9bc12be61c8"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.056947 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:04.072247 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:04.076550 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:04.076838 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:04.095067 5983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45049
I20260620 18:30:04.095116 6049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45049 every 8 connection(s)
I20260620 18:30:04.096437 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:04.100242 6050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:04.105899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 5983
I20260620 18:30:04.106115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:04.106933 388 external_mini_cluster.cc:1428] Setting key c8f2daaed7bb3c4eaf61ea7221cf3ea3
I20260620 18:30:04.112717 6050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6: Bootstrap starting.
I20260620 18:30:04.116185 6050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:04.117276 6050 log.cc:824] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:04.119937 6050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6: No bootstrap required, opened a new log
Jun 20 18:30:04 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:04.129137 6050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260620 18:30:04.129492 6050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:04.129598 6050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5337b1264e014d9cb5052c907677c0f6, State: Initialized, Role: FOLLOWER
I20260620 18:30:04.130224 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:04.109210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.97.62:45049)
0620 18:30:04.110052 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:04.110083 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:04.110124 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:30:04.111904 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:04.111927 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:04.112308 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:04.112689 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:04.114047 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.115374 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.116690 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.117363 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.117538 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:04.122255 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:04.122263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:04.122277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:04.122318 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:04.125778 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:04.126582 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:04.126589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:04.126595 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:04.126761 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:04.127181 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:04.127184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:04.127187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:04.127843 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:04.128089 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:04.128490 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 18:30:04.128867 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":402,"threads_started":1}
I20260620 18:30:04.130203 6050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260620 18:30:04.130436 6050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:04.130610 6050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:04.130800 6050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:04.133771 6050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260620 18:30:04.134224 6050 leader_election.cc:302] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5337b1264e014d9cb5052c907677c0f6; no voters:
I20260620 18:30:04.134791 6050 leader_election.cc:288] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:04.134923 6055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:04.135623 6055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [term 1 LEADER]: Becoming Leader. State: Replica: 5337b1264e014d9cb5052c907677c0f6, State: Running, Role: LEADER
I20260620 18:30:04.136186 6055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260620 18:30:04.136788 6050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:04.139701 6056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5337b1264e014d9cb5052c907677c0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260620 18:30:04.139593 6057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5337b1264e014d9cb5052c907677c0f6. Latest consensus state: current_term: 1 leader_uuid: "5337b1264e014d9cb5052c907677c0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5337b1264e014d9cb5052c907677c0f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260620 18:30:04.140450 6056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:04.140448 6057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:04.144783 6063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:04.149322 6063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:04.158254 6063 catalog_manager.cc:1347] Generated new cluster ID: 19e0d47b50ec4324959fb388006b7e59
I20260620 18:30:04.158387 6063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:04.166047 6063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:04.167301 6063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:04.178699 6063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5337b1264e014d9cb5052c907677c0f6: Generated new TSK 0
I20260620 18:30:04.179488 6063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:30:04.230779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:04.575317 6076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:04.575614 6076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:04.575711 6076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:04.582724 6076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:04.582827 6076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:04.582937 6076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:04.592257 6076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:04.593423 6076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:04.594640 6076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:04.607394 6082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:04.607338 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:04.607431 6081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:04.607973 6076 server_base.cc:1034] running on GCE node
I20260620 18:30:04.608657 6076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:04.609472 6076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:04.610664 6076 hybrid_clock.cc:648] HybridClock initialized: now 1781980204610627 us; error 39 us; skew 500 ppm
I20260620 18:30:04.610924 6076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:04 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:04.614090 6076 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:30:04.615561 6076 webserver.cc:460] Webserver started at http://127.0.97.1:43085/ using document root <none> and password file <none>
I20260620 18:30:04.616190 6076 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:04.616285 6076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:04.616509 6076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:04.619351 6076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "bbaa0b6a50ab41969eabfe09e96dbeaf"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "9be5460080e36cdfe9a19168e3bd7a37"
server_key_iv: "66c734d3f89479c640f388dc65b51079"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.620148 6076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bbaa0b6a50ab41969eabfe09e96dbeaf"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "9be5460080e36cdfe9a19168e3bd7a37"
server_key_iv: "66c734d3f89479c640f388dc65b51079"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.625126 6076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:30:04.628343 6091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:04.629925 6076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:30:04.630093 6076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "bbaa0b6a50ab41969eabfe09e96dbeaf"
format_stamp: "Formatted at 2026-06-20 18:30:04 on dist-test-slave-mcnm"
server_key: "9be5460080e36cdfe9a19168e3bd7a37"
server_key_iv: "66c734d3f89479c640f388dc65b51079"
server_key_version: "encryptionkey@0"
I20260620 18:30:04.630262 6076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:04.657474 6076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:04.661782 6076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:04.662014 6076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:04.662972 6076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:04.664300 6076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:04.664371 6076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:04.664450 6076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:04.664494 6076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:04.697606 6076 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43677
I20260620 18:30:04.697695 6203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43677 every 8 connection(s)
I20260620 18:30:04.698853 6076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:04.703315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6076
I20260620 18:30:04.703536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:04.704383 388 external_mini_cluster.cc:1428] Setting key b1cf6c2aaac946f5c38bbb42c997501d
Jun 20 18:30:04 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:04.721894 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:04.701471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54063 (local address 127.0.97.62:45049)
0620 18:30:04.701935 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:04.701939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:04.702912 (+ 973us) server_negotiation.cc:407] Connection header received
0620 18:30:04.705514 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:04.705520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:04.705598 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:04.705715 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:04.707839 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.708548 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.711106 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.711644 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.711764 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:04.715923 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:04.715930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:04.715941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:04.715993 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:04.718776 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:04.719566 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:04.719571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:04.719576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:04.719700 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:04.720153 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:04.720158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:04.720162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:04.720421 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:04.720576 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:04.721302 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 18:30:04.721519 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260620 18:30:04.722957 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:04.701971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.1:54063)
0620 18:30:04.702691 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:04.702743 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:04.705181 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:04.705959 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:04.705968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:04.706788 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:04.707646 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:04.707676 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.708725 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.708734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:04.710919 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:04.710943 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:04.711881 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:04.711890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:04.712221 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:04.713048 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:04.713070 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:04.715636 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:04.718922 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:04.718933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:04.718949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:04.719390 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:04.719815 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:04.719821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:04.719825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:04.720034 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:04.720549 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:04.720557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:04.720910 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 18:30:04.721199 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 18:30:04.721743 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":281,"threads_started":1}
I20260620 18:30:04.724788 6204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260620 18:30:04.725188 6204 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:04.726233 6204 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:30:04.728729 6015 ts_manager.cc:194] Registered new tserver with Master: bbaa0b6a50ab41969eabfe09e96dbeaf (127.0.97.1:43677)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 18:30:04.730461 6015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:30:04.754135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:05.073196 6210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:05.073469 6210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:05.073563 6210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:05.080487 6210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:05.080575 6210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:05.080798 6210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:05.089998 6210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:05.091197 6210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:05.092545 6210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:05.107692 6215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:05.107649 6216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:05.107826 6218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:05.108982 6210 server_base.cc:1034] running on GCE node
I20260620 18:30:05.109707 6210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:05.110882 6210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:05.112109 6210 hybrid_clock.cc:648] HybridClock initialized: now 1781980205112043 us; error 57 us; skew 500 ppm
I20260620 18:30:05.112489 6210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.116314 6210 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:30:05.117707 6210 webserver.cc:460] Webserver started at http://127.0.97.2:34543/ using document root <none> and password file <none>
I20260620 18:30:05.118366 6210 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:05.118469 6210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:05.118721 6210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:05.121757 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "fc179e6537784cc2aea4a8e9ac143c9c"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "261e32f690a05a49b47562dcb79ca2e6"
server_key_iv: "f5b8cb58ad42d602ab3da1c1dac595be"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.122529 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fc179e6537784cc2aea4a8e9ac143c9c"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "261e32f690a05a49b47562dcb79ca2e6"
server_key_iv: "f5b8cb58ad42d602ab3da1c1dac595be"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.127811 6210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 18:30:05.131204 6225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.132680 6210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:05.132822 6210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "fc179e6537784cc2aea4a8e9ac143c9c"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "261e32f690a05a49b47562dcb79ca2e6"
server_key_iv: "f5b8cb58ad42d602ab3da1c1dac595be"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.133041 6210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:05.150542 6210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:05.154449 6210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:05.154682 6210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:05.155503 6210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:05.156929 6210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:05.157008 6210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.157092 6210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:05.157135 6210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.188266 6210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45497
I20260620 18:30:05.188311 6337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45497 every 8 connection(s)
I20260620 18:30:05.189493 6210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:05.197098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6210
I20260620 18:30:05.197290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:05.197851 388 external_mini_cluster.cc:1428] Setting key 0c3418dcba8a70639e5f48f69db688cc
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:05.209456 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.191851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58841 (local address 127.0.97.62:45049)
0620 18:30:05.192046 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.192052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.193255 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 18:30:05.194862 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.194866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.194928 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.195020 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.196409 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.197072 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.199116 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.199801 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.199895 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.203998 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.204003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.204006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.204036 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.206560 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.207268 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.207274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.207278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.207402 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.207838 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.207845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.207848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.208104 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.208267 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.208902 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.209127 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:30:05.210354 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.192396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.2:58841)
0620 18:30:05.193106 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:05.193147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:05.194636 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:05.195215 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:05.195223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:05.195760 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:05.196274 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:05.196289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.197230 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.197236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:05.198975 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:05.198985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.200014 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.200022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:05.200327 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.201089 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:05.201123 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:05.203775 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:05.206713 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:05.206722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:05.206737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:05.207140 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:05.207544 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:05.207549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:05.207553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:05.207727 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:05.208229 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:05.208237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:05.208562 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 18:30:05.208811 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:30:05.209287 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":270,"threads_started":1}
I20260620 18:30:05.211795 6338 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260620 18:30:05.212131 6338 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:05.212829 6338 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260620 18:30:05.215274 6015 ts_manager.cc:194] Registered new tserver with Master: fc179e6537784cc2aea4a8e9ac143c9c (127.0.97.2:45497)
I20260620 18:30:05.216403 6015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58841
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:30:05.246742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:05.565243 6343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:05.565555 6343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:05.565719 6343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:05.573428 6343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:05.573571 6343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:05.573751 6343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:05.583356 6343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:05.584645 6343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:05.585964 6343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:05.598680 6348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:05.598703 6351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:05.598762 6349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:05.599498 6343 server_base.cc:1034] running on GCE node
I20260620 18:30:05.600090 6343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:05.600888 6343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:05.602128 6343 hybrid_clock.cc:648] HybridClock initialized: now 1781980205602064 us; error 54 us; skew 500 ppm
I20260620 18:30:05.602501 6343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.605945 6343 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:30:05.607422 6343 webserver.cc:460] Webserver started at http://127.0.97.3:45281/ using document root <none> and password file <none>
I20260620 18:30:05.608150 6343 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:05.608258 6343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:05.608520 6343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:05.611436 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "735b76fec76b4cc98fc828b01dd9c9f4"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "e980724e9f4413de0b7a73ed772092ef"
server_key_iv: "806e1467c91ca4a0cd71d41b681a0415"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.612270 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "735b76fec76b4cc98fc828b01dd9c9f4"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "e980724e9f4413de0b7a73ed772092ef"
server_key_iv: "806e1467c91ca4a0cd71d41b681a0415"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.617517 6343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:30:05.620819 6358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.622144 6343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:30:05.622370 6343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "735b76fec76b4cc98fc828b01dd9c9f4"
format_stamp: "Formatted at 2026-06-20 18:30:05 on dist-test-slave-mcnm"
server_key: "e980724e9f4413de0b7a73ed772092ef"
server_key_iv: "806e1467c91ca4a0cd71d41b681a0415"
server_key_version: "encryptionkey@0"
I20260620 18:30:05.622601 6343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:05.637970 6343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:05.641928 6343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:05.642233 6343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:05.643208 6343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:05.644906 6343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:05.644999 6343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.645177 6343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:05.645238 6343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:05.679086 6343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33525
I20260620 18:30:05.679121 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33525 every 8 connection(s)
I20260620 18:30:05.680465 6343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:05.687183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6343
I20260620 18:30:05.687357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:05.687832 388 external_mini_cluster.cc:1428] Setting key c3aa5864b56e39f4215059c75d0ab8c5
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:05.700692 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.682915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60361 (local address 127.0.97.62:45049)
0620 18:30:05.683115 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.683120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.684355 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 18:30:05.686201 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.686207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.686305 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.686432 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.688113 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.689227 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.691034 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.691409 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.691558 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.695105 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.695109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.695113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.695142 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.697558 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.698437 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.698441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.698443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.698552 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.699005 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.699008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.699010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.699240 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.699340 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.700173 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.700415 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:30:05.701856 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.683380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.3:60361)
0620 18:30:05.684150 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:05.684187 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:05.685921 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:05.686668 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:05.686677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:05.687179 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:05.687901 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:05.687942 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.689393 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.689399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:05.690871 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:05.690884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.691670 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.691680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:05.691916 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.692524 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:05.692548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:05.694913 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:05.697697 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:05.697705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:05.697720 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:05.698321 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:05.698647 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:05.698652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:05.698656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:05.698893 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:05.699359 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:05.699366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:05.699769 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 18:30:05.700073 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 18:30:05.700591 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260620 18:30:05.703414 6471 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260620 18:30:05.703737 6471 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:05.704388 6471 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260620 18:30:05.705741 6015 ts_manager.cc:194] Registered new tserver with Master: 735b76fec76b4cc98fc828b01dd9c9f4 (127.0.97.3:33525)
I20260620 18:30:05.706591 6015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60361
I20260620 18:30:05.715261 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:30:05.733428 6204 heartbeater.cc:503] Master 127.0.97.62:45049 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/test-admin.keytab.
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.741761 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:30:05.747047 6478 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:05.744513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.0.97.1:43677)
0620 18:30:05.745181 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.745189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.745214 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:30:05.745267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.745273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.745488 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.745657 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.746575 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38376: BlockingRecv error: recv got EOF from 127.0.0.1:38376 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":292,"threads_started":1}
W20260620 18:30:05.747170 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.748788 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.750115 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:30:05.765703 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.751786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.0.97.1:43677)
0620 18:30:05.751978 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.751982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.752072 (+ 90us) server_negotiation.cc:407] Connection header received
0620 18:30:05.752218 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.752223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.752282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.752364 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.753329 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.754262 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.755616 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.756183 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.756285 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.759879 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.759890 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.759905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.759945 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.762826 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.763598 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.763605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.763607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.763720 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.764129 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.764137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.764139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.764520 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.764746 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.765119 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.765429 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 18:30:05.771448 6478 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:05.769800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.97.1:43677)
0620 18:30:05.770033 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.770039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.770215 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:30:05.770326 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.770330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.770387 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.770463 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.771182 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38390: BlockingRecv error: recv got EOF from 127.0.0.1:38390 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 18:30:05.771478 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.772927 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.774186 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:30:05.788185 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.775596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.0.97.1:43677)
0620 18:30:05.775743 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.775751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.776116 (+ 365us) server_negotiation.cc:407] Connection header received
0620 18:30:05.776294 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.776305 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.776358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.776454 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.777328 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.778244 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.779765 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.780202 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.780296 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.783556 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.783582 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.783588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.783625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.786101 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.786686 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.786690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.786692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.786799 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.787111 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.787114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.787116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.787348 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.787588 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.787825 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.787967 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 18:30:05.792237 6478 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:05.790738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.97.1:43677)
0620 18:30:05.790944 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.790953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.791013 (+ 60us) server_negotiation.cc:407] Connection header received
0620 18:30:05.791149 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.791153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.791238 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.791341 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.791985 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38422: BlockingRecv error: recv got EOF from 127.0.0.1:38422 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:30:05.792331 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980152255989-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.793843 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:05.795086 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:30:05.808898 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.796409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.0.97.1:43677)
0620 18:30:05.796628 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.796635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.796858 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:30:05.797044 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.797047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.797097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.797193 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.798112 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.798797 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.800077 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.800486 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.800618 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.804052 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.804057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.804061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.804089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.806460 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.807080 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.807093 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.807096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.807233 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.807611 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.807618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.807621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.807866 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.808196 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.808405 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.808598 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:30:05.810308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6076
I20260620 18:30:05.821724 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.810984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.0.97.1:43677)
0620 18:30:05.811169 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.811178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.811443 (+ 265us) server_negotiation.cc:407] Connection header received
0620 18:30:05.811632 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.811635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.811683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.811755 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.812640 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.813267 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.814433 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.814824 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.814910 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.816788 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.816793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.816796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.816823 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.819194 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.819858 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.819862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.819864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.819985 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.820372 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.820377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.820379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.820678 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.820892 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.821226 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.821440 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:30:05.822368 6199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:05.905438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6076
I20260620 18:30:05.921967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6210
Jun 20 18:30:05 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:30:05.938727 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:05.922979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.97.2:45497)
0620 18:30:05.923601 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:05.923609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:05.923634 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:30:05.923698 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:05.923705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:05.923925 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:05.924058 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:05.925040 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.926106 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.927439 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:05.928183 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:05.928298 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:05.932275 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:05.932291 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:05.932306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:05.932345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:05.935450 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.936277 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.936289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.936299 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.936436 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:05.936910 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:05.936918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:05.936921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:05.937399 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:05.937723 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:05.938103 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 18:30:05.938393 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":333,"threads_started":1}
I20260620 18:30:05.940151 6333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:06.020475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6210
I20260620 18:30:06.036798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6343
Jun 20 18:30:06 dist-test-slave-mcnm krb5kdc[5970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:30:06.052508 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:06.037708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.0.97.3:33525)
0620 18:30:06.038354 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:06.038361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:06.038390 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:30:06.038445 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:06.038453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:06.038664 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:06.038790 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:06.039877 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:06.040915 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:06.042123 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:06.042670 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:06.042767 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:06.047092 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:06.047102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:06.047116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:06.047146 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:06.049962 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:06.050655 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:06.050661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:06.050664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:06.050768 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:06.051115 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:06.051118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:06.051120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:06.051411 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:06.051737 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:06.051971 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 18:30:06.052213 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260620 18:30:06.053767 6466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:06.145329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6343
I20260620 18:30:06.161101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 5983
I20260620 18:30:06.162288 6045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:06.268026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 5983
2026-06-20T18:30:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3916 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:06 dist-test-slave-mcnm krb5kdc[6498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:06 dist-test-slave-mcnm krb5kdc[6498](info): set up 2 sockets
Jun 20 18:30:06 dist-test-slave-mcnm krb5kdc[6498](info): commencing operation
krb5kdc: starting...
W20260620 18:30:07.352205 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:07 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 18:30:07.557178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45127
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:07.868134 6511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:07.868423 6511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:07.868587 6511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:07.875635 6511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:07.875743 6511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:07.875826 6511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:07.875914 6511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:07.876000 6511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:07.885561 6511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:07.886878 6511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:07.888221 6511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:07.899892 6517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:07.899930 6519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:07.899992 6516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:07.900478 6511 server_base.cc:1034] running on GCE node
I20260620 18:30:07.901292 6511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:07.902731 6511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:07.903923 6511 hybrid_clock.cc:648] HybridClock initialized: now 1781980207903885 us; error 45 us; skew 500 ppm
I20260620 18:30:07.904268 6511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:07 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:07.907701 6511 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 18:30:07.909312 6511 webserver.cc:460] Webserver started at http://127.0.97.62:43067/ using document root <none> and password file <none>
I20260620 18:30:07.909997 6511 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:07.910142 6511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:07.910455 6511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:07.913450 6511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "340bd78a35364e1cbc026f88541efd29"
format_stamp: "Formatted at 2026-06-20 18:30:07 on dist-test-slave-mcnm"
server_key: "0a36d048f486eb005f65d7f5c91928a2"
server_key_iv: "3517117a6219686d5ecafc82e9de2a5b"
server_key_version: "encryptionkey@0"
I20260620 18:30:07.914211 6511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "340bd78a35364e1cbc026f88541efd29"
format_stamp: "Formatted at 2026-06-20 18:30:07 on dist-test-slave-mcnm"
server_key: "0a36d048f486eb005f65d7f5c91928a2"
server_key_iv: "3517117a6219686d5ecafc82e9de2a5b"
server_key_version: "encryptionkey@0"
I20260620 18:30:07.919351 6511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:30:07.922832 6526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:07.924327 6511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:07.924510 6511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "340bd78a35364e1cbc026f88541efd29"
format_stamp: "Formatted at 2026-06-20 18:30:07 on dist-test-slave-mcnm"
server_key: "0a36d048f486eb005f65d7f5c91928a2"
server_key_iv: "3517117a6219686d5ecafc82e9de2a5b"
server_key_version: "encryptionkey@0"
I20260620 18:30:07.924705 6511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:07.948240 6511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:07.952684 6511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:07.952979 6511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:07.972015 6511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45127
I20260620 18:30:07.972056 6577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45127 every 8 connection(s)
I20260620 18:30:07.973538 6511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:07.976807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6511
I20260620 18:30:07.977010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:07.977663 6578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:07.977833 388 external_mini_cluster.cc:1428] Setting key 201cfa62deacc12a754ffddfe3330288
I20260620 18:30:07.989902 6578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Bootstrap starting.
Jun 20 18:30:07 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:07.993235 6578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:07.994297 6578 log.cc:824] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:07.997087 6578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: No bootstrap required, opened a new log
I20260620 18:30:07.999300 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:07.979867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.0.97.62:45127)
0620 18:30:07.980612 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:07.980624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:07.980667 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:30:07.982424 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:07.982445 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:07.982836 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:07.983231 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:07.984674 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:07.985910 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:07.987098 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:07.987883 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:07.988087 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:07.992192 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:07.992200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:07.992214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:07.992249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:07.995438 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:07.996129 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:07.996135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:07.996141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:07.996306 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:07.996680 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:07.996684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:07.996687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:07.997188 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:07.997493 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:07.997914 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 18:30:07.998335 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":346,"threads_started":1}
I20260620 18:30:08.005138 6578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } }
I20260620 18:30:08.005596 6578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:08.005724 6578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bd78a35364e1cbc026f88541efd29, State: Initialized, Role: FOLLOWER
I20260620 18:30:08.006330 6578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } }
I20260620 18:30:08.006572 6578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:08.006711 6578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:08.006883 6578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:08.009665 6578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } }
I20260620 18:30:08.010188 6578 leader_election.cc:302] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 340bd78a35364e1cbc026f88541efd29; no voters:
I20260620 18:30:08.010766 6578 leader_election.cc:288] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:08.010921 6583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:08.011649 6583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 LEADER]: Becoming Leader. State: Replica: 340bd78a35364e1cbc026f88541efd29, State: Running, Role: LEADER
I20260620 18:30:08.012200 6583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } }
I20260620 18:30:08.012889 6578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:08.015321 6585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bd78a35364e1cbc026f88541efd29. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } }
I20260620 18:30:08.015813 6585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:08.015332 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } }
I20260620 18:30:08.017076 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:08.020151 6589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:08.025077 6589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:08.035307 6589 catalog_manager.cc:1347] Generated new cluster ID: e6b6a77882c94f9b8017a00e1c872585
I20260620 18:30:08.035451 6589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:08.079387 6589 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:08.080716 6589 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:08.096818 6589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Generated new TSK 0
I20260620 18:30:08.097667 6589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 18:30:08.154659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:08.477160 6604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:08.477432 6604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:08.477593 6604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:08.484817 6604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:08.484923 6604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:08.485133 6604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:08.494777 6604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:08.496102 6604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:08.497401 6604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:08.510713 6610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:08.510958 6612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:08.511193 6609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:08.511667 6604 server_base.cc:1034] running on GCE node
I20260620 18:30:08.512319 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:08.513399 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:08.514611 6604 hybrid_clock.cc:648] HybridClock initialized: now 1781980208514555 us; error 38 us; skew 500 ppm
I20260620 18:30:08.514897 6604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:08 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:08.518129 6604 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 18:30:08.519618 6604 webserver.cc:460] Webserver started at http://127.0.97.1:44041/ using document root <none> and password file <none>
I20260620 18:30:08.520249 6604 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:08.520346 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:08.520587 6604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:08.524005 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
format_stamp: "Formatted at 2026-06-20 18:30:08 on dist-test-slave-mcnm"
server_key: "83a5071acff2512951a5f6542705d486"
server_key_iv: "0782b2b431ebc7cee403ddb1c5f9ff0d"
server_key_version: "encryptionkey@0"
I20260620 18:30:08.524886 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
format_stamp: "Formatted at 2026-06-20 18:30:08 on dist-test-slave-mcnm"
server_key: "83a5071acff2512951a5f6542705d486"
server_key_iv: "0782b2b431ebc7cee403ddb1c5f9ff0d"
server_key_version: "encryptionkey@0"
I20260620 18:30:08.530174 6604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:08.533595 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:08.534924 6604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 18:30:08.535137 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
format_stamp: "Formatted at 2026-06-20 18:30:08 on dist-test-slave-mcnm"
server_key: "83a5071acff2512951a5f6542705d486"
server_key_iv: "0782b2b431ebc7cee403ddb1c5f9ff0d"
server_key_version: "encryptionkey@0"
I20260620 18:30:08.535344 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:08.577701 6604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:08.582007 6604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:08.582337 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:08.583302 6604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:08.584767 6604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:08.584870 6604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:08.585024 6604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:08.585103 6604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:08.617587 6604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37701
I20260620 18:30:08.617630 6731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37701 every 8 connection(s)
I20260620 18:30:08.618834 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:08.624663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6604
I20260620 18:30:08.624871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:08.625489 388 external_mini_cluster.cc:1428] Setting key a98f2d30e5d87b037b8fdc7e0d2ffeac
Jun 20 18:30:08 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:08.642499 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:08.621429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42027 (local address 127.0.97.62:45127)
0620 18:30:08.621881 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:08.621887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:08.622882 (+ 995us) server_negotiation.cc:407] Connection header received
0620 18:30:08.624887 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:08.624893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:08.625001 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:08.625124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:08.626762 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:08.627789 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:08.630492 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:08.630879 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:08.630976 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:08.635227 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:08.635233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:08.635236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:08.635267 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:08.638856 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:08.639885 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:08.639894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:08.639901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:08.640046 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:08.640482 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:08.640490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:08.640496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:08.640878 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:08.641027 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:08.641765 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 18:30:08.642017 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
I20260620 18:30:08.643165 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:08.622008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:42027)
0620 18:30:08.622702 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:08.622762 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:08.624613 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:08.625360 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:08.625369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:08.625927 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:08.626565 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:08.626588 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:08.627999 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:08.628005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:08.630324 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:08.630341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:08.631111 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:08.631120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:08.631440 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:08.632185 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:08.632219 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:08.634997 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:08.639031 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:08.639039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:08.639055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:08.639762 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:08.640163 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:08.640167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:08.640169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:08.640363 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:08.641034 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:08.641040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:08.641373 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 18:30:08.641603 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 18:30:08.642018 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":290,"threads_started":1}
I20260620 18:30:08.644837 6732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:08.645135 6732 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:08.645844 6732 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:30:08.647940 6543 ts_manager.cc:194] Registered new tserver with Master: a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:08.649487 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 18:30:08.678678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:08.999231 6738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:08.999480 6738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:08.999662 6738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:09.006723 6738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:09.006825 6738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:09.007062 6738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:09.016672 6738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:09.017922 6738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:09.019243 6738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:09.033067 6743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:09.033214 6744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:09.033630 6746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:09.033952 6738 server_base.cc:1034] running on GCE node
I20260620 18:30:09.034628 6738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:09.035742 6738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:09.037011 6738 hybrid_clock.cc:648] HybridClock initialized: now 1781980209036927 us; error 64 us; skew 500 ppm
I20260620 18:30:09.037392 6738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:09 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980209, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:09.043922 6738 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 18:30:09.045425 6738 webserver.cc:460] Webserver started at http://127.0.97.2:35833/ using document root <none> and password file <none>
I20260620 18:30:09.046116 6738 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:09.046244 6738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:09.046572 6738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:09.049515 6738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f0ae4d34fac454995aa3508ec23357e"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "3b82932c62220ca1288af79ef766f866"
server_key_iv: "37dbb7fcf7df0cd55ca13a0035de070b"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.050273 6738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f0ae4d34fac454995aa3508ec23357e"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "3b82932c62220ca1288af79ef766f866"
server_key_iv: "37dbb7fcf7df0cd55ca13a0035de070b"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.055244 6738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:30:09.058662 6753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.060139 6738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:09.060317 6738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "3f0ae4d34fac454995aa3508ec23357e"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "3b82932c62220ca1288af79ef766f866"
server_key_iv: "37dbb7fcf7df0cd55ca13a0035de070b"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.060523 6738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:09.084419 6738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:09.088415 6738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:09.088717 6738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:09.089658 6738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:09.090956 6738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:09.091037 6738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.091207 6738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:09.091274 6738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.125415 6738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33521
I20260620 18:30:09.125460 6865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33521 every 8 connection(s)
I20260620 18:30:09.126693 6738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:09.128438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6738
I20260620 18:30:09.128633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:09.129081 388 external_mini_cluster.cc:1428] Setting key 11a8b9064808268b02a0ddb4dd4cd24c
Jun 20 18:30:09 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980209, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:09.149784 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:09.130240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50145 (local address 127.0.97.62:45127)
0620 18:30:09.130432 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:09.130438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:09.131819 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 18:30:09.133600 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:09.133608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:09.133706 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:09.133815 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:09.135220 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.136390 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.138522 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.139383 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.139531 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:09.143105 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:09.143114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:09.143122 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:09.143175 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:09.146779 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:09.147456 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:09.147464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:09.147469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:09.147642 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:09.148016 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:09.148022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:09.148026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:09.148396 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:09.148542 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:09.149181 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 18:30:09.149428 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:30:09.150492 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:09.130814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:50145)
0620 18:30:09.131653 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:09.131711 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:09.133349 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:09.133999 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:09.134008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:09.134518 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:09.135045 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:09.135062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.136561 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.136569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:09.138390 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:09.138401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.139633 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.139639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:09.139888 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:09.140558 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:09.140580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:09.142869 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:09.146914 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:09.146921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:09.146933 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:09.147337 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:09.147742 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:09.147745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:09.147748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:09.147910 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:09.148523 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:09.148529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:09.148812 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 18:30:09.149030 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 18:30:09.149398 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":346,"threads_started":1}
I20260620 18:30:09.151939 6866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:09.152256 6866 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260620 18:30:09.153046 6866 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
I20260620 18:30:09.154481 6543 ts_manager.cc:194] Registered new tserver with Master: 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:09.155269 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 18:30:09.176832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:09.496179 6871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:09.496443 6871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:09.496536 6871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:09.503826 6871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:09.503935 6871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:09.504051 6871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:09.514298 6871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:09.515499 6871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:09.516836 6871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:09.530216 6876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:09.530300 6879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:09.530331 6877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:09.530853 6871 server_base.cc:1034] running on GCE node
I20260620 18:30:09.531409 6871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:09.532348 6871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:09.533591 6871 hybrid_clock.cc:648] HybridClock initialized: now 1781980209533518 us; error 50 us; skew 500 ppm
I20260620 18:30:09.533924 6871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:09 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980209, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:09.536975 6871 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 18:30:09.538269 6871 webserver.cc:460] Webserver started at http://127.0.97.3:34689/ using document root <none> and password file <none>
I20260620 18:30:09.538940 6871 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:09.539042 6871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:09.539287 6871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:09.542179 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "41a08f8b79534940a66de768c7ca02a5"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "7fba4a160a28b5b9a1d53963c7026aa8"
server_key_iv: "db7d1f524b72269edb77f015aabe5f6d"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.542970 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "41a08f8b79534940a66de768c7ca02a5"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "7fba4a160a28b5b9a1d53963c7026aa8"
server_key_iv: "db7d1f524b72269edb77f015aabe5f6d"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.548167 6871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:30:09.551442 6886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.552757 6871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:30:09.552925 6871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "41a08f8b79534940a66de768c7ca02a5"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "7fba4a160a28b5b9a1d53963c7026aa8"
server_key_iv: "db7d1f524b72269edb77f015aabe5f6d"
server_key_version: "encryptionkey@0"
I20260620 18:30:09.553112 6871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:09.605310 6871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:09.609397 6871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:09.609699 6871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:09.610682 6871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:09.612023 6871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:09.612126 6871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.612272 6871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:09.612347 6871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:09.649507 6871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44153
I20260620 18:30:09.649541 6998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44153 every 8 connection(s)
I20260620 18:30:09.651144 6871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:09.652436 6732 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
I20260620 18:30:09.656220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 6871
I20260620 18:30:09.656387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:09.656864 388 external_mini_cluster.cc:1428] Setting key 5590603c20029f938bff1349ed284082
Jun 20 18:30:09 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980209, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:09.672012 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:09.654291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52979 (local address 127.0.97.62:45127)
0620 18:30:09.654772 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:09.654781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:09.655318 (+ 537us) server_negotiation.cc:407] Connection header received
0620 18:30:09.657743 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:09.657748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:09.657855 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:09.657982 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:09.659534 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.660404 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.662064 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.662487 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.662634 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:09.666253 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:09.666258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:09.666262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:09.666293 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:09.668915 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:09.669645 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:09.669653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:09.669657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:09.669787 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:09.670210 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:09.670216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:09.670220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:09.670515 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:09.670673 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:09.671358 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 18:30:09.671614 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260620 18:30:09.672941 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:09.654370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:52979)
0620 18:30:09.655167 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:09.655207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:09.657444 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:09.658228 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:09.658236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:09.658772 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:09.659336 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:09.659356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.660572 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.660578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:09.661904 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:09.661915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:09.662733 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:09.662740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:09.663000 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:09.663762 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:09.663785 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:09.666051 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:09.669057 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:09.669068 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:09.669085 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:09.669505 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:09.669898 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:09.669904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:09.669909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:09.670092 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:09.670656 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:09.670664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:09.670991 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:30:09.671266 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:30:09.671826 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":317,"threads_started":1}
I20260620 18:30:09.674314 6999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:09.674578 6999 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:09.675258 6999 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
I20260620 18:30:09.676668 6543 ts_manager.cc:194] Registered new tserver with Master: 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153)
I20260620 18:30:09.677425 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52979
I20260620 18:30:09.683681 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 18:30:09.730671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35353
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45127,127.0.97.61:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:10.050742 7005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:10.051014 7005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:10.051111 7005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:10.058326 7005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:10.058421 7005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:10.058468 7005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:10.058525 7005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:10.058573 7005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:10.068362 7005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45127,127.0.97.61:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
W20260620 18:30:10.068655 7005 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45127,127.0.97.61:35353'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:30:10.069615 7005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:10.070854 7005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:10.082371 7010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.082326 7011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.082343 7013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:10.082832 7005 server_base.cc:1034] running on GCE node
I20260620 18:30:10.083952 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:10.085239 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:10.086458 7005 hybrid_clock.cc:648] HybridClock initialized: now 1781980210086400 us; error 34 us; skew 500 ppm
I20260620 18:30:10.086822 7005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:10.089951 7005 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 18:30:10.091285 7005 webserver.cc:460] Webserver started at http://127.0.97.61:44013/ using document root <none> and password file <none>
I20260620 18:30:10.091979 7005 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:10.092062 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:10.092308 7005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:10.095024 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/instance:
uuid: "39f59276d8d74d2f891944e602f2c4ac"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "53429a0ba9c2dedf2c1b12faf12bd288"
server_key_iv: "1a79cc497f2e5d57b1f3eea462855a03"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.095795 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal/instance:
uuid: "39f59276d8d74d2f891944e602f2c4ac"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "53429a0ba9c2dedf2c1b12faf12bd288"
server_key_iv: "1a79cc497f2e5d57b1f3eea462855a03"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.100618 7005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:30:10.103976 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:10.105448 7005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:30:10.105650 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "39f59276d8d74d2f891944e602f2c4ac"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "53429a0ba9c2dedf2c1b12faf12bd288"
server_key_iv: "1a79cc497f2e5d57b1f3eea462855a03"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.105868 7005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:10.129623 7005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:10.133939 7005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:10.134158 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:10.152364 7005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35353
I20260620 18:30:10.152382 7071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35353 every 8 connection(s)
I20260620 18:30:10.153637 7005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
I20260620 18:30:10.157294 7072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:10.157647 6866 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
I20260620 18:30:10.159349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7005
I20260620 18:30:10.167147 7072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:10.182762 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.161530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.97.61:35353)
0620 18:30:10.162298 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.162310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.162351 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:30:10.163914 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.163936 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.164421 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.164757 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.166513 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.167868 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.169527 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.170654 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.171341 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.175411 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.175418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.175428 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.175463 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.178799 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.179451 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.179457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.179463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.179680 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.180130 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.180134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.180136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.180651 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.180901 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.181337 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.181672 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":376,"threads_started":1}
I20260620 18:30:10.184743 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.169772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.97.62:45127)
0620 18:30:10.169922 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.169926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.170753 (+ 827us) server_negotiation.cc:407] Connection header received
0620 18:30:10.170968 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.170973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.171078 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.171183 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.172685 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.173353 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.174994 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.175379 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.175553 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.179116 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.179121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.179124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.179154 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.182095 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.182756 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.182763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.182769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.182929 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.183280 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.183285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.183288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.183644 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.183788 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.184215 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.184432 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:30:10.184836 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.169823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38436)
0620 18:30:10.170589 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.170630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.170831 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.171231 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.171240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.172179 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.172518 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.172531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.173491 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.173497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.174852 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.174863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.175597 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.175604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.175779 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.176574 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.176594 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.178920 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.182214 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.182226 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.182233 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.182638 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.183042 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.183047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.183050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.183165 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.183784 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.183791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.183943 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.184131 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.184413 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":432,"threads_started":1}
I20260620 18:30:10.186314 7072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 18:30:10.197786 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.187095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44652)
0620 18:30:10.187286 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.187305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.187420 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.187700 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.187704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.188031 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.188141 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.188148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.189020 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.189028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.189849 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.189858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.190488 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.190493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.190694 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.191141 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.191157 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.193128 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.195847 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.195850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.195853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.196216 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.196526 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.196530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.196532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.196663 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.197096 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.197108 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.197226 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.197338 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.197521 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":36}
I20260620 18:30:10.197871 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.187111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.97.61:35353)
0620 18:30:10.187272 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.187282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.187370 (+ 88us) server_negotiation.cc:407] Connection header received
0620 18:30:10.187548 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.187552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.187598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.187686 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.188259 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.188905 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.189985 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.190349 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.190482 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.193278 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.193283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.193286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.193311 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.195742 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.196311 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.196314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.196316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.196414 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.196752 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.196755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.196757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.196984 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.197091 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.197409 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.197632 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":49}
I20260620 18:30:10.202247 7072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Bootstrap starting.
I20260620 18:30:10.205214 7072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:10.206163 7072 log.cc:824] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:10.208654 7072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: No bootstrap required, opened a new log
I20260620 18:30:10.215680 7072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } }
I20260620 18:30:10.215953 7072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:10.216027 7072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f59276d8d74d2f891944e602f2c4ac, State: Initialized, Role: FOLLOWER
I20260620 18:30:10.216485 7072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } }
I20260620 18:30:10.217187 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } } }
I20260620 18:30:10.217516 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:30:10.218087 7072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:30:10.226907 7089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:30:10.227047 7089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:30:10.228951 7005 master_runner.cc:428] Detected that this master 39f59276d8d74d2f891944e602f2c4ac is joining an existing cluster
I20260620 18:30:10.228868 7078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:10.229089 7005 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:35353
I20260620 18:30:10.229200 7078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } }
I20260620 18:30:10.230513 7078 leader_election.cc:288] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127)
I20260620 18:30:10.231169 6553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f59276d8d74d2f891944e602f2c4ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340bd78a35364e1cbc026f88541efd29" is_pre_election: true
I20260620 18:30:10.232918 7023 leader_election.cc:302] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 39f59276d8d74d2f891944e602f2c4ac; no voters: 340bd78a35364e1cbc026f88541efd29
I20260620 18:30:10.233724 7078 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:30:10.258721 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.234464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.0.97.61:35353)
0620 18:30:10.234745 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.234752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.243574 (+ 8822us) server_negotiation.cc:407] Connection header received
0620 18:30:10.244160 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.244168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.244253 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.244333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.246237 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.247404 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.249284 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.250050 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.250173 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.252371 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.252380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.252386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.252422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.255441 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.256297 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.256305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.256311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.256517 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.256938 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.256945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.256951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.257367 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.257499 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.258018 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.258256 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":81}
I20260620 18:30:10.260227 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.233781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44668)
0620 18:30:10.242959 (+ 9178us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.242985 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.243979 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.244574 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.244582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.245540 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.246053 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.246066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.247649 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.247657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.249133 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.249148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.250245 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.250251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.250414 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.251106 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.251123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.252187 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.255649 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.255656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.255663 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.256184 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.256652 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.256657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.256662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.256825 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.257578 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.257581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.257744 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.259324 (+ 1580us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.259691 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8989,"thread_start_us":13900,"threads_started":1}
I20260620 18:30:10.262192 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.235178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.0.97.62:45127)
0620 18:30:10.235361 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.235368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.248103 (+ 12735us) server_negotiation.cc:407] Connection header received
0620 18:30:10.248275 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.248280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.248357 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.248470 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.249634 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.250501 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.252611 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.253236 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.253333 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.255984 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.255993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.255998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.256038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.259304 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.260106 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.260112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.260116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.260270 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.260712 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.260717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.260721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.261036 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.261190 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.261618 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.261825 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:30:10.262214 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.235500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38448)
0620 18:30:10.247982 (+ 12482us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.248000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.248146 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.248526 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.248531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.249149 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.249491 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.249499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.250672 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.250679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.252433 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.252447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.253650 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.253657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.253855 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.254538 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.254559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.255770 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.259471 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.259476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.259480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.259965 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.260414 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.260420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.260423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.260580 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.261143 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.261148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.261334 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.261473 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.261761 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"mutex_wait_us":23,"thread_start_us":8042,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 18:30:10.285351 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.269240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.0.97.61:35353)
0620 18:30:10.270012 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.270018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.270036 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:10.270128 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.270133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.270238 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.270336 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.271788 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.272847 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.274340 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.275139 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.275279 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.280051 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.280062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.280068 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.280095 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.282715 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.283357 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.283363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.283366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.283495 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.283888 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.283896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.283899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.284151 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.284303 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.284777 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.285003 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667}
I20260620 18:30:10.285424 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.268996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44682)
0620 18:30:10.269654 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.269698 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.269863 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.270405 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.270414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.271284 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.271564 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.271577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.272974 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.272978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.274195 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.274211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.275313 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.275320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.275577 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.276964 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.276988 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.279872 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.282845 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.282857 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.282865 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.283237 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.283667 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.283670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.283672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.283786 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.284279 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.284286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.284493 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.284678 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.284972 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":288,"threads_started":1}
I20260620 18:30:10.287322 6543 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:35353: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 35353
}
attrs {
promote: true
}
}
dest_uuid: "340bd78a35364e1cbc026f88541efd29"
cas_config_opid_index: -1
I20260620 18:30:10.288252 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } }
I20260620 18:30:10.294998 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } }
I20260620 18:30:10.295861 7047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:10.298802 7047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:30:10.299279 7101 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:35353
I20260620 18:30:10.299805 7101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:10.300201 7005 master.cc:537] Master@127.0.97.61:35353 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 18:30:10.301939 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } }
I20260620 18:30:10.302265 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:10.303710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34305
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:10.307307 6527 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 18:30:10.313876 7005 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:10.314422 7005 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:10.314654 7005 tablet_replica.cc:330] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: stopping tablet replica
W20260620 18:30:10.321666 6527 consensus_peers.cc:487] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 -> Peer 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61:35353): Couldn't send request to peer 39f59276d8d74d2f891944e602f2c4ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:30:10.323153 7005 master.cc:552] Master@127.0.97.61:35353 shutdown complete.
I20260620 18:30:10.323246 7005 master_runner.cc:305] Clearing existing system tablet
I20260620 18:30:10.323693 7005 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:10.323798 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:10.326185 7005 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 18:30:10.327728 7108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:10.328212 7005 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:10.328645 7005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 18:30:10.328738 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "39f59276d8d74d2f891944e602f2c4ac"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "53429a0ba9c2dedf2c1b12faf12bd288"
server_key_iv: "1a79cc497f2e5d57b1f3eea462855a03"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.332468 7005 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:10.337054 7005 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:30:10.337249 7005 log.cc:1196] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:30:10.337666 7005 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Deleting consensus metadata
I20260620 18:30:10.337916 7005 master_runner.cc:315] Copying system tablet from 127.0.97.62:45127
I20260620 18:30:10.340559 7005 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45127
I20260620 18:30:10.359067 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.341283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.0.97.62:45127)
0620 18:30:10.341491 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.341501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.348071 (+ 6570us) server_negotiation.cc:407] Connection header received
0620 18:30:10.348290 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.348295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.348369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.348473 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.349272 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.350032 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.351217 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.351710 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.351843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.353597 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.353602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.353608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.353646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.356441 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.357141 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.357148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.357152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.357270 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.357661 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.357667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.357671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.357996 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.358148 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.358524 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.358715 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:30:10.359172 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.341260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38462)
0620 18:30:10.347925 (+ 6665us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.347956 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.348131 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.348501 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.348506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.348914 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.349110 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.349119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.350194 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.350200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.351071 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.351083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.351861 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.351867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.352057 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.352698 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.352715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.353399 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.356573 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.356578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.356583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.357023 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.357413 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.357417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.357421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.357555 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.358121 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.358126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.358284 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.358431 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.358712 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6530,"thread_start_us":6442,"threads_started":1}
I20260620 18:30:10.360297 6563 tablet_copy_service.cc:140] P 340bd78a35364e1cbc026f88541efd29: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 39f59276d8d74d2f891944e602f2c4ac ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:38462)
I20260620 18:30:10.360528 6563 tablet_copy_service.cc:161] P 340bd78a35364e1cbc026f88541efd29: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 39f59276d8d74d2f891944e602f2c4ac at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:38462: session id = 39f59276d8d74d2f891944e602f2c4ac-00000000000000000000000000000000
I20260620 18:30:10.363397 6563 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:30:10.365347 7005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:30:10.369683 7005 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: tablet copy: Starting download of 0 data blocks...
I20260620 18:30:10.369974 7005 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: tablet copy: Starting download of 1 WAL segments...
I20260620 18:30:10.372273 7005 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:30:10.375268 6563 tablet_copy_service.cc:342] P 340bd78a35364e1cbc026f88541efd29: Request end of tablet copy session 39f59276d8d74d2f891944e602f2c4ac-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:38462
I20260620 18:30:10.375428 6563 tablet_copy_service.cc:433] P 340bd78a35364e1cbc026f88541efd29: ending tablet copy session 39f59276d8d74d2f891944e602f2c4ac-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 39f59276d8d74d2f891944e602f2c4ac
I20260620 18:30:10.384722 7005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:10.388701 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.396206 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.404750 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:10.408841 7005 server_base.cc:1034] running on GCE node
I20260620 18:30:10.409200 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:10.409837 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:10.411053 7005 hybrid_clock.cc:648] HybridClock initialized: now 1781980210410990 us; error 31 us; skew 500 ppm
I20260620 18:30:10.411273 7005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:10.413204 7005 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 18:30:10.414551 7005 webserver.cc:460] Webserver started at http://127.0.97.61:34709/ using document root <none> and password file <none>
I20260620 18:30:10.414865 7005 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:10.414959 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:10.417356 7005 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:30:10.418924 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:10.419339 7005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:30:10.419464 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "39f59276d8d74d2f891944e602f2c4ac"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "53429a0ba9c2dedf2c1b12faf12bd288"
server_key_iv: "1a79cc497f2e5d57b1f3eea462855a03"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.419669 7005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:10.434803 7005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:10.438364 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:10.478502 7005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35353
I20260620 18:30:10.478502 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35353 every 8 connection(s)
I20260620 18:30:10.479370 7005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
I20260620 18:30:10.486218 7177 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:30:10.491499 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Bootstrap starting.
I20260620 18:30:10.503254 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:30:10.503772 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Bootstrap complete.
I20260620 18:30:10.504585 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } }
I20260620 18:30:10.504804 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39f59276d8d74d2f891944e602f2c4ac, State: Initialized, Role: LEARNER
I20260620 18:30:10.505074 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } }
I20260620 18:30:10.505815 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } }
I20260620 18:30:10.506157 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:10.506192 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: LEARNER
I20260620 18:30:10.515264 7191 catalog_manager.cc:1259] Loaded cluster ID: e6b6a77882c94f9b8017a00e1c872585
I20260620 18:30:10.515401 7191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: loading cluster ID for follower catalog manager: success
I20260620 18:30:10.518808 7191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: acquiring CA information for follower catalog manager: success
I20260620 18:30:10.521116 7191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:30:10.680250 6999 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
W20260620 18:30:10.698594 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:10.698884 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:10.699066 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:10.706699 7106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:10.706866 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:10.706960 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:10.707077 7106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:10.707131 7106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:10.718029 7106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:10.719432 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:10.720919 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:10.733269 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.733515 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:10.733603 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:10.734757 7106 server_base.cc:1034] running on GCE node
I20260620 18:30:10.735626 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:10.736991 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:10.738221 7106 hybrid_clock.cc:648] HybridClock initialized: now 1781980210738180 us; error 41 us; skew 500 ppm
I20260620 18:30:10.738593 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:10.741812 7106 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 18:30:10.743211 7106 webserver.cc:460] Webserver started at http://127.0.97.60:40295/ using document root <none> and password file <none>
I20260620 18:30:10.743961 7106 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:10.744108 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:10.744406 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:10.747243 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/instance:
uuid: "1774525a63a343869db365001e3b30fd"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "146d19f41d41aebb6cc85663d9a153cb"
server_key_iv: "35c0cbb832356a7c6014dbcf62584352"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.748051 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal/instance:
uuid: "1774525a63a343869db365001e3b30fd"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "146d19f41d41aebb6cc85663d9a153cb"
server_key_iv: "35c0cbb832356a7c6014dbcf62584352"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.753333 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:10.757571 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:10.759007 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:10.759203 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "1774525a63a343869db365001e3b30fd"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "146d19f41d41aebb6cc85663d9a153cb"
server_key_iv: "35c0cbb832356a7c6014dbcf62584352"
server_key_version: "encryptionkey@0"
I20260620 18:30:10.759438 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:10.775056 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:10.780364 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:10.780678 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:10.800110 7106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34305
I20260620 18:30:10.800165 7257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34305 every 8 connection(s)
I20260620 18:30:10.801484 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
I20260620 18:30:10.803540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7106
I20260620 18:30:10.806787 7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:10.815935 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:10.834180 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.805813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.97.60:34305)
0620 18:30:10.806656 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.806667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.806708 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:30:10.808594 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.808616 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.809020 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.809403 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.810806 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.812904 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.814287 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.815378 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.815588 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.821308 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.821318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.821330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.821373 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.825913 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.827115 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.827129 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.827143 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.827490 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.828162 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.828170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.828177 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.829332 (+ 1155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.831324 (+ 1992us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.831742 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.832317 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":361,"threads_started":1}
I20260620 18:30:10.840368 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.818768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.97.62:45127)
0620 18:30:10.818992 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.819000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.819702 (+ 702us) server_negotiation.cc:407] Connection header received
0620 18:30:10.820123 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.820131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.820209 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.820296 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.821809 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.822986 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.826144 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.826972 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.827119 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.832327 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.832337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.832350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.832404 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.836736 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.837950 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.837958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.837965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.838179 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.838646 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.838649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.838651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.838988 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.839163 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.839771 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.840013 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:30:10.840436 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.818663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38472)
0620 18:30:10.819549 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.819596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.819851 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.820399 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.820408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.821319 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.821624 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.821634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.823159 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.823167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.825904 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.825926 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.827399 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.827409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.827671 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.828489 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.828520 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.832060 (+ 3540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.837000 (+ 4940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.837015 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.837026 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.837760 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.838324 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.838332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.838338 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.838511 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.839169 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.839177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.839383 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.839622 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.839963 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":428,"threads_started":1}
I20260620 18:30:10.842355 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:30:10.845327 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.824064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44688)
0620 18:30:10.838626 (+ 14562us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.838648 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.838778 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.839273 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.839276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.839297 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:10.839486 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.839493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.841087 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.841091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.842836 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.842848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.844547 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.844553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.844730 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.844801 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.844909 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.845043 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14434,"thread_start_us":14342,"threads_started":1}
I20260620 18:30:10.845837 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.824268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.0.97.61:35353)
0620 18:30:10.838203 (+ 13935us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.838208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.838724 (+ 516us) server_negotiation.cc:407] Connection header received
0620 18:30:10.838907 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.838912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.838988 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.839143 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:10.839691 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.840948 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.843042 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.844378 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.844521 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.844734 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.844936 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.845082 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13779,"thread_start_us":13654,"threads_started":1}
I20260620 18:30:10.847234 7152 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 18:30:10.847885 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bd78a35364e1cbc026f88541efd29. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } }
I20260620 18:30:10.848160 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: LEARNER
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 18:30:10.856797 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.843041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44692)
0620 18:30:10.843229 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.843250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.843381 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.844025 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.844029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.844382 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.844511 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.844519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.846333 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.846339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.847413 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.847426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.848423 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.848428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.848563 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.849173 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.849189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.851155 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.854187 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.854192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.854197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.854588 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.855005 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.855010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.855015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.855155 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.855757 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.855762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.855946 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.856094 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.856400 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:30:10.856832 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.843138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.97.61:35353)
0620 18:30:10.843723 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.843729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.843749 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:30:10.843793 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.843798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.843871 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.843984 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.844688 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.846176 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.847644 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.848235 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.848393 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.851358 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.851363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.851366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.851396 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.854022 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.854718 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.854726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.854730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.854880 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.855269 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.855277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.855280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.855631 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.855777 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.856206 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.856417 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":273,"threads_started":1}
I20260620 18:30:10.858017 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980210, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 18:30:10.870322 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.858798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.0.1:33320)
0620 18:30:10.859109 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.859123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.859215 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.859482 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.859487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.859863 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.860032 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.860041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.860959 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.860962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.861808 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.861818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.862444 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.862448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.863068 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.863647 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.863663 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.865810 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.868444 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.868448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.868451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.868777 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.869101 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.869104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.869106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.869236 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.869646 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.869649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.869754 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.869856 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.870065 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260620 18:30:10.870481 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.858788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.0.97.60:34305)
0620 18:30:10.858966 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.858971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.859211 (+ 240us) server_negotiation.cc:407] Connection header received
0620 18:30:10.859306 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.859310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.859365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.859445 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.860156 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.860856 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.861931 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.862315 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.862403 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.865993 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.865997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.866001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.866028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.868334 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.868870 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.868874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.868876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.869011 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.869321 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.869324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.869326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.869555 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.869654 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.869963 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.870169 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":78}
I20260620 18:30:10.875000 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Bootstrap starting.
I20260620 18:30:10.878439 7258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:10.879740 7258 log.cc:824] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:10.882779 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: No bootstrap required, opened a new log
I20260620 18:30:10.892920 7258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } }
I20260620 18:30:10.893260 7258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:10.893391 7258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1774525a63a343869db365001e3b30fd, State: Initialized, Role: FOLLOWER
I20260620 18:30:10.894069 7258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } }
I20260620 18:30:10.895010 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } } }
I20260620 18:30:10.895422 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:30:10.896173 7258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:30:10.906612 7279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:30:10.906780 7279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:30:10.911197 7106 master_runner.cc:428] Detected that this master 1774525a63a343869db365001e3b30fd is joining an existing cluster
I20260620 18:30:10.911325 7106 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:34305
I20260620 18:30:10.941015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6604
I20260620 18:30:10.947958 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.915768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.97.60:34305)
0620 18:30:10.916510 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.916519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.928453 (+ 11934us) server_negotiation.cc:407] Connection header received
0620 18:30:10.929958 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.929963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.930041 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.930127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.931392 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.932599 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.935305 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.936262 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.936355 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.939328 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.939336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.939340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.939373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.944799 (+ 5426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.945743 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.945749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.945753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.945947 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.946321 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.946325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.946329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.946688 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.946821 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.947344 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.947575 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":595}
I20260620 18:30:10.949196 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.916156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.0.1:33326)
0620 18:30:10.928318 (+ 12162us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.928353 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.928513 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.930253 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.930258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.930877 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.931249 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.931261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.932778 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.932783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.935106 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.935120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.936812 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.936864 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.937041 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.937910 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.937932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.939126 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.945008 (+ 5882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.945015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.945020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.945620 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.946078 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.946082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.946086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.946214 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.946920 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.946924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.947109 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.948448 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.948750 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12020,"thread_start_us":11829,"threads_started":1}
I20260620 18:30:10.950423 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.916705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.0.97.62:45127)
0620 18:30:10.916980 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.916989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.923290 (+ 6301us) server_negotiation.cc:407] Connection header received
0620 18:30:10.924035 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.924045 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.924153 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.924241 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.926175 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.927298 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.929964 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.930693 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.930804 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.935052 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.935061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.935068 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.935116 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.941748 (+ 6632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.948088 (+ 6340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.948095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.948098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.948260 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.948774 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.948781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.948787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.949126 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.949256 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.949837 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.950084 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:30:10.951339 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.915743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38480)
0620 18:30:10.923141 (+ 7398us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.923167 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.923827 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.924534 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.924542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.925510 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.925978 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.925992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.927903 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.927908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.929778 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.929796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.931132 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.931141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.931375 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.933530 (+ 2155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.933557 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.934804 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.941907 (+ 7103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.941913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.941917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.945441 (+ 3524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.948406 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.948412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.948416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.948618 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.949331 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.949335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.949550 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.950582 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.950905 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7187,"mutex_wait_us":5,"thread_start_us":10567,"threads_started":1}
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 18:30:10.959956 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.915733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.61:35353)
0620 18:30:10.916329 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.916336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.926296 (+ 9960us) server_negotiation.cc:407] Connection header received
0620 18:30:10.926748 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.926754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.926848 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.926916 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.928624 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.929726 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.945938 (+ 16212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.946612 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.946715 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.953084 (+ 6369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.953092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.953097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.953137 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.956959 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.957722 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.957729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.957735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.957904 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.958339 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.958344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.958347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.958663 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.958787 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.959357 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.959599 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260620 18:30:10.960899 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.916129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44698)
0620 18:30:10.926204 (+ 10075us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.926221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.926590 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.927093 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.927097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.928084 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.928488 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.928499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.939649 (+ 11150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.939654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.945754 (+ 6100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.945766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.947050 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.947057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.947230 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.951785 (+ 4555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.951807 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.952861 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.957108 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.957114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.957118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.957587 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.958037 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.958043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.958047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.958205 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.958883 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.958888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.959073 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.960124 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.960442 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9915,"thread_start_us":7197,"threads_started":1}
I20260620 18:30:10.963783 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.942122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.0.97.1:37701)
0620 18:30:10.942821 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.942832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.942866 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:30:10.942941 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.942952 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.943261 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.944907 (+ 1646us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.945196 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.946765 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.948609 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.949488 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.955180 (+ 5692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.955486 (+ 306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.955501 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.955555 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.955595 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.959567 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.960478 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.960486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.960490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.960636 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.961207 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.961212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.961216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.961627 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.962895 (+ 1268us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.963058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.963364 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":731,"threads_started":1}
I20260620 18:30:10.965126 6727 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 18:30:10 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 18:30:10.974377 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.957512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.0.1:33342)
0620 18:30:10.957695 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:10.957715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:10.957812 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:10.958412 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:10.958418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:10.958844 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:10.959037 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.959049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.960883 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.960888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.962205 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:10.962218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.963578 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.963585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:10.963770 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.964514 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:10.964536 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:10.967610 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:10.971398 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.971403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.971407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.971985 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.972459 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:10.972463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:10.972467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:10.972624 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:10.973335 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:10.973340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:10.973519 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 18:30:10.973670 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:30:10.973974 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:30:10.975426 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:10.957845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.97.60:34305)
0620 18:30:10.958014 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:10.958020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:10.958036 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:30:10.958126 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:10.958133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:10.958236 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:10.959187 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:10.959353 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.960721 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.962588 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:10.963327 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:10.963601 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:10.967808 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:10.967817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:10.967827 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:10.967872 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:10.971247 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.972129 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.972179 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.972183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.972325 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:10.972757 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:10.972763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:10.972767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:10.973124 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:10.973256 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:10.974791 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0620 18:30:10.975064 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:30:10.976773 6543 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:34305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1774525a63a343869db365001e3b30fd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 34305
}
attrs {
promote: true
}
}
dest_uuid: "340bd78a35364e1cbc026f88541efd29"
cas_config_opid_index: 5
I20260620 18:30:10.978482 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } }
I20260620 18:30:10.980422 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1774525a63a343869db365001e3b30fd (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:10.993098 7233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:10.993392 7263 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:34305
I20260620 18:30:10.997915 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:10.998260 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:30:10.998485 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:10.998939 7263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:30:11.007141 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1774525a63a343869db365001e3b30fd (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.010712 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: true } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.011001 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: LEARNER
I20260620 18:30:11.011445 7291 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: attempting to promote NON_VOTER 39f59276d8d74d2f891944e602f2c4ac to VOTER
I20260620 18:30:11.012588 7291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } }
I20260620 18:30:11.015141 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 LEARNER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:30:11.015682 7291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:30:11.017866 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.019847 7106 master.cc:537] Master@127.0.97.60:34305 shutting down...
I20260620 18:30:11.019941 7233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:30:11.020788 7232 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:30:11.020977 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.021193 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.021420 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.021584 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.022226 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:11.023329 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.027673 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.027935 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:30:11.033118 7212 tablet.cc:2369] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:11.035763 7106 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:11.036226 7106 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:11.036468 7106 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: stopping tablet replica
I20260620 18:30:11.044101 7106 master.cc:552] Master@127.0.97.60:34305 shutdown complete.
I20260620 18:30:11.044199 7106 master_runner.cc:305] Clearing existing system tablet
I20260620 18:30:11.044716 7106 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:11.044837 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 18:30:11.047230 6530 consensus_peers.cc:487] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 -> Peer 1774525a63a343869db365001e3b30fd (127.0.97.60:34305): Couldn't send request to peer 1774525a63a343869db365001e3b30fd. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:30:11.052173 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.007s
I20260620 18:30:11.053601 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:11.053921 7106 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:11.054200 7106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:30:11.054313 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "1774525a63a343869db365001e3b30fd"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "146d19f41d41aebb6cc85663d9a153cb"
server_key_iv: "35c0cbb832356a7c6014dbcf62584352"
server_key_version: "encryptionkey@0"
I20260620 18:30:11.057997 7106 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:11.062336 7106 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:30:11.062510 7106 log.cc:1196] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:30:11.062928 7106 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Deleting consensus metadata
I20260620 18:30:11.063175 7106 master_runner.cc:315] Copying system tablet from 127.0.97.62:45127
I20260620 18:30:11.065696 7106 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45127
I20260620 18:30:11.083099 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.066415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.0.97.62:45127)
0620 18:30:11.066678 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.066688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.072022 (+ 5334us) server_negotiation.cc:407] Connection header received
0620 18:30:11.072229 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.072235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.072308 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.072413 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:11.073236 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.074000 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.075482 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.075986 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.076122 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.077899 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:11.077906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:11.077911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:11.077951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:11.080612 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.081227 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.081235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.081240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.081390 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.081749 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.081756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.081759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.081995 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:11.082152 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.082495 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.082722 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 18:30:11.083426 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.066377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38496)
0620 18:30:11.071886 (+ 5509us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:11.071910 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:11.072083 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:11.072460 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:11.072464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:11.072896 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:11.073080 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.073088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.074134 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.074141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.075331 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.075342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.076132 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.076137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.076316 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.076980 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:11.076997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:11.077702 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:11.080741 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.080749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.080753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.081088 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.081501 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.081506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.081510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.081647 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.082107 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:11.082112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:11.082268 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:30:11.082406 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:30:11.082720 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5379,"thread_start_us":5296,"threads_started":1}
I20260620 18:30:11.084007 6563 tablet_copy_service.cc:140] P 340bd78a35364e1cbc026f88541efd29: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1774525a63a343869db365001e3b30fd ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38496)
I20260620 18:30:11.084192 6563 tablet_copy_service.cc:161] P 340bd78a35364e1cbc026f88541efd29: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1774525a63a343869db365001e3b30fd at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38496: session id = 1774525a63a343869db365001e3b30fd-00000000000000000000000000000000
I20260620 18:30:11.086370 6563 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:30:11.088078 7106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:30:11.092726 7106 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: tablet copy: Starting download of 0 data blocks...
I20260620 18:30:11.093016 7106 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: tablet copy: Starting download of 1 WAL segments...
I20260620 18:30:11.095016 7106 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:30:11.098241 6563 tablet_copy_service.cc:342] P 340bd78a35364e1cbc026f88541efd29: Request end of tablet copy session 1774525a63a343869db365001e3b30fd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38496
I20260620 18:30:11.098369 6563 tablet_copy_service.cc:433] P 340bd78a35364e1cbc026f88541efd29: ending tablet copy session 1774525a63a343869db365001e3b30fd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1774525a63a343869db365001e3b30fd
I20260620 18:30:11.108091 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:11.110899 7304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:11.113046 7305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:11.120299 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:11.121641 7106 server_base.cc:1034] running on GCE node
I20260620 18:30:11.121948 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:11.122624 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:11.122677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6604
I20260620 18:30:11.124233 7106 hybrid_clock.cc:648] HybridClock initialized: now 1781980211124192 us; error 34 us; skew 500 ppm
I20260620 18:30:11.124500 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980211, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:11.130069 7106 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 18:30:11.131340 7106 webserver.cc:460] Webserver started at http://127.0.97.60:41425/ using document root <none> and password file <none>
I20260620 18:30:11.131699 7106 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:11.131800 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:11.135123 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 18:30:11.137171 7314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:11.137594 7106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:30:11.137717 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "1774525a63a343869db365001e3b30fd"
format_stamp: "Formatted at 2026-06-20 18:30:10 on dist-test-slave-mcnm"
server_key: "146d19f41d41aebb6cc85663d9a153cb"
server_key_iv: "35c0cbb832356a7c6014dbcf62584352"
server_key_version: "encryptionkey@0"
I20260620 18:30:11.137920 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:11.144767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37701
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45127
--webserver_port=44041
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:30:11.155090 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:11.159548 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:11.195061 7106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34305
I20260620 18:30:11.195111 7366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34305 every 8 connection(s)
I20260620 18:30:11.196035 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
I20260620 18:30:11.201134 7367 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:30:11.205047 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Bootstrap starting.
I20260620 18:30:11.221768 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:30:11.222219 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: Bootstrap complete.
I20260620 18:30:11.223208 7367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } }
I20260620 18:30:11.223486 7367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1774525a63a343869db365001e3b30fd, State: Initialized, Role: LEARNER
I20260620 18:30:11.223750 7367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } }
I20260620 18:30:11.224537 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.224799 7367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:11.224942 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: LEARNER
I20260620 18:30:11.263473 7381 catalog_manager.cc:1259] Loaded cluster ID: e6b6a77882c94f9b8017a00e1c872585
I20260620 18:30:11.263689 7381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: loading cluster ID for follower catalog manager: success
I20260620 18:30:11.267191 7381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: acquiring CA information for follower catalog manager: success
I20260620 18:30:11.269436 7381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:30:11.407970 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.401138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.97.60:34305)
0620 18:30:11.401658 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.401663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.401679 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:30:11.401737 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.401740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.401813 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.401922 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:11.402409 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.403396 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.406000 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.406848 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.406986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.407221 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.407554 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.407676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260620 18:30:11.408036 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.400967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.0.1:33350)
0620 18:30:11.401235 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:11.401255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:11.401388 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:11.401984 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:11.401992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:11.402038 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:11.402262 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.402275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.403557 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.403566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.405791 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.405808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.407043 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.407053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.407314 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.407360 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:30:11.407462 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:30:11.407685 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 18:30:11.409138 7342 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 18:30:11.409739 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bd78a35364e1cbc026f88541efd29. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.410029 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: LEARNER
W20260620 18:30:11.508910 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:11.509213 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:11.509379 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:11.516784 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:11.516939 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:11.517103 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:11.527750 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44041
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:11.529282 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:11.530700 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:11.543812 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:11.543880 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:11.543896 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:11.549166 7319 server_base.cc:1034] running on GCE node
I20260620 18:30:11.549767 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:11.550696 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:11.551935 7319 hybrid_clock.cc:648] HybridClock initialized: now 1781980211551878 us; error 32 us; skew 500 ppm
I20260620 18:30:11.552223 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980211, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:11.555802 7319 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 18:30:11.557463 7319 webserver.cc:460] Webserver started at http://127.0.97.1:44041/ using document root <none> and password file <none>
I20260620 18:30:11.558113 7319 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:11.558233 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:11.563555 7319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:11.566243 7397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:11.567687 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:11.567812 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
format_stamp: "Formatted at 2026-06-20 18:30:08 on dist-test-slave-mcnm"
server_key: "83a5071acff2512951a5f6542705d486"
server_key_iv: "0782b2b431ebc7cee403ddb1c5f9ff0d"
server_key_version: "encryptionkey@0"
I20260620 18:30:11.568730 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:11.581351 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:11.585516 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:11.585737 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:11.586547 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:11.587884 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:11.587968 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:11.588034 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:11.588086 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:11.622964 7319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37701
I20260620 18:30:11.623054 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37701 every 8 connection(s)
I20260620 18:30:11.624372 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:11.630643 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7319
I20260620 18:30:11.630971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6738
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980211, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980211, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980211, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 18:30:11.653250 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.629129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57165 (local address 127.0.97.60:34305)
0620 18:30:11.629293 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.629299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.629669 (+ 370us) server_negotiation.cc:407] Connection header received
0620 18:30:11.632646 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.632655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.632762 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.632848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:11.635694 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.637122 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.640506 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.640972 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.641046 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.646023 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:11.646033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:11.646041 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:11.646088 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:11.649905 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.650709 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.650716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.650720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.650848 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.651334 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.651339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.651342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.651685 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:11.651810 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.652666 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.652891 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 18:30:11 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 18:30:11.655695 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.628352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.97.1:57165)
0620 18:30:11.629500 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:11.629562 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:11.632295 (+ 2733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:11.633485 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:11.633500 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:11.634428 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:11.635427 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.635468 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.637386 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.637395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.640337 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.640346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.641311 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.641315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.641502 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.642261 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:11.642296 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:11.645762 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:11.650029 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.650038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.650054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.650571 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.650989 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.650994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.650998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.651210 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.651894 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:11.651901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:11.652289 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 18:30:11.653510 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0620 18:30:11.654169 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":968,"threads_started":1}
I20260620 18:30:11.656463 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.628352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:58127)
0620 18:30:11.636001 (+ 7649us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:11.636025 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:11.636171 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:11.636897 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:11.636904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:11.637701 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:11.638148 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.638162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.640486 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.640491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.642065 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.642082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.643430 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.643435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.643665 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.644907 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:11.644935 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:11.648776 (+ 3841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:11.652661 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.652666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.652670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.653212 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.653695 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.653700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.653703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.653846 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.654848 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:11.654854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:11.655000 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 18:30:11.655146 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:30:11.655875 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7466,"mutex_wait_us":215,"spinlock_wait_cycles":14208,"thread_start_us":4311,"threads_started":1}
I20260620 18:30:11.657047 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.627200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58127 (local address 127.0.97.62:45127)
0620 18:30:11.629135 (+ 1935us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.629143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.636301 (+ 7158us) server_negotiation.cc:407] Connection header received
0620 18:30:11.636387 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.636396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.636508 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.636627 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:11.639172 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.640302 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.642278 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.643015 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.643154 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.649005 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:11.649012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:11.649017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:11.649059 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:11.652502 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.653376 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.653381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.653385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.653556 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.653982 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.653988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.653991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.654637 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:11.654780 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.656240 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.656466 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1691,"threads_started":1}
I20260620 18:30:11.657444 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34305
I20260620 18:30:11.657788 7510 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:11.657990 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:11.658109 7511 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:11.658732 7510 heartbeater.cc:511] Master 127.0.97.60:34305 requested a full tablet report, sending...
I20260620 18:30:11.658757 7511 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
I20260620 18:30:11.661475 6543 ts_manager.cc:194] Re-registered known tserver with Master: a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:11.664286 7332 ts_manager.cc:194] Registered new tserver with Master: a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:11.664381 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.632213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.2:33521)
0620 18:30:11.639945 (+ 7732us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.639953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.639980 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:30:11.640043 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.640050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.640276 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.643810 (+ 3534us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:11.644019 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.645991 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.647973 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.649042 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.649701 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.654949 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:11.654963 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:11.654978 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:11.655021 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:11.659032 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.659951 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.659962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.659967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.660146 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.661040 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.661046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.661049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.661470 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:11.661769 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.663561 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.663885 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7567,"thread_start_us":8224,"threads_started":1}
I20260620 18:30:11.666363 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.628352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.97.1:37131)
0620 18:30:11.629713 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:11.629731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:11.638377 (+ 8646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:11.639016 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:11.639021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:11.639614 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:11.640327 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.640339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.641812 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.641819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.643538 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:11.643558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.644524 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.644534 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:11.644695 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.645216 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:11.645240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:11.648890 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:11.661615 (+ 12725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.661620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.661624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.663448 (+ 1824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.663993 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:11.663998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:11.664002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:11.664157 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:11.665103 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:11.665110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:11.665289 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 18:30:11.665440 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:30:11.665728 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":41,"spinlock_wait_cycles":47488,"thread_start_us":7136,"threads_started":1}
I20260620 18:30:11.666395 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58127
I20260620 18:30:11.666611 6861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:11.667317 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:11.628999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37131 (local address 127.0.97.61:35353)
0620 18:30:11.638743 (+ 9744us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:11.638749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:11.638781 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:30:11.638836 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:11.638840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:11.638908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:11.638986 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:11.640480 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.641651 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.643731 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:11.644342 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:11.644469 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:11.649167 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:11.649173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:11.649177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:11.649215 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:11.661418 (+ 12203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.663644 (+ 2226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.663650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.663654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.663858 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:11.664310 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:11.664317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:11.664320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:11.664650 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:11.664795 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:11.666655 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0620 18:30:11.666916 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9602,"thread_start_us":9496,"threads_started":1}
I20260620 18:30:11.670017 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35353
I20260620 18:30:11.670174 7512 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:11.670678 7512 heartbeater.cc:511] Master 127.0.97.61:35353 requested a full tablet report, sending...
I20260620 18:30:11.673171 7142 ts_manager.cc:194] Registered new tserver with Master: a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:11.770077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6738
I20260620 18:30:11.785926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33521
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45127
--webserver_port=35833
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:30:11.949231 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: true } } }
I20260620 18:30:11.949539 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.950999 7524 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: attempting to promote NON_VOTER 1774525a63a343869db365001e3b30fd to VOTER
I20260620 18:30:11.951709 7524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:11.953725 7342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 LEARNER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:30:11.953794 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Refusing update from remote peer 340bd78a35364e1cbc026f88541efd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:30:11.954383 7523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:30:11.954813 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:30:11.957743 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1774525a63a343869db365001e3b30fd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.958550 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1774525a63a343869db365001e3b30fd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.960711 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.960927 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.961592 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.961833 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.962267 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.962494 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:11.962926 7342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1774525a63a343869db365001e3b30fd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.966367 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.966738 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:30:11.967196 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bd78a35364e1cbc026f88541efd29" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:11.967550 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:30:12.115358 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:12.115723 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:12.115821 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:12.122877 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:12.122997 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:12.123121 7522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:12.133263 7522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35833
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:12.134562 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:12.135923 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:12.148752 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:12.148936 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:12.148988 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:12.150102 7522 server_base.cc:1034] running on GCE node
I20260620 18:30:12.150703 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:12.151664 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:12.152923 7522 hybrid_clock.cc:648] HybridClock initialized: now 1781980212152877 us; error 37 us; skew 500 ppm
I20260620 18:30:12.153268 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:12.156711 7522 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 18:30:12.158226 7522 webserver.cc:460] Webserver started at http://127.0.97.2:35833/ using document root <none> and password file <none>
I20260620 18:30:12.158828 7522 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:12.158941 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:12.164077 7522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:30:12.166780 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.168288 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:12.168424 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "3f0ae4d34fac454995aa3508ec23357e"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "3b82932c62220ca1288af79ef766f866"
server_key_iv: "37dbb7fcf7df0cd55ca13a0035de070b"
server_key_version: "encryptionkey@0"
I20260620 18:30:12.169530 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:12.182216 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:12.185899 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:12.186124 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:12.186977 7522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:12.188278 7522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:12.188364 7522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.188434 7522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:12.188480 7522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.227968 7522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33521
I20260620 18:30:12.227993 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33521 every 8 connection(s)
I20260620 18:30:12.229486 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:12.235957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7522
I20260620 18:30:12.236263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6871
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:12.285354 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.232251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57617 (local address 127.0.97.61:35353)
0620 18:30:12.246692 (+ 14441us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.246697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.263852 (+ 17155us) server_negotiation.cc:407] Connection header received
0620 18:30:12.264022 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.264030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.264154 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.264303 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.267418 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.268930 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.271667 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.272322 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.272397 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.277434 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.277441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.277447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.277487 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.281576 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.282583 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.282591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.282596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.282739 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.283247 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.283253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.283258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.283681 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.283803 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.284733 (+ 930us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.284965 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14331,"thread_start_us":14408,"threads_started":1}
I20260620 18:30:12.287617 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.234018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.97.2:57617)
0620 18:30:12.263708 (+ 29690us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.263729 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.263878 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.264896 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.264914 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.266076 (+ 1162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.267101 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.267132 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.269180 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.269189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.271445 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.271460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.272774 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.272780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.273039 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.273861 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.273896 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.277172 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.281726 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.281737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.281756 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.282441 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.282892 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.282897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.282901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.283099 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.283891 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.283898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.284285 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.285652 (+ 1367us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.286178 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29544,"thread_start_us":29369,"threads_started":1}
I20260620 18:30:12.287920 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.232897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.97.2:60635)
0620 18:30:12.259672 (+ 26775us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.259742 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.262856 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.271001 (+ 8145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.271006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.271606 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.272014 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.272026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.273732 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.273737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.275373 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.275386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.277262 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.277268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.277445 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.278059 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.278082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.280771 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.285130 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.285138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.285144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.285607 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.286073 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.286083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.286259 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.286924 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.286928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.287058 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.287174 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.287439 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26362,"mutex_wait_us":3,"thread_start_us":36652,"threads_started":1}
I20260620 18:30:12.288880 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.232269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48391 (local address 127.0.97.62:45127)
0620 18:30:12.270538 (+ 38269us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.270544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.270563 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:30:12.270615 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.270619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.270705 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.270820 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.273523 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.274621 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.276646 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.277275 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.277386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.281531 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.281539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.281545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.281595 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.285103 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286112 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.286118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.286123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.286253 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286651 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.286656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.286659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.286979 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.287104 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.288277 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.288503 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38154,"thread_start_us":38066,"threads_started":1}
I20260620 18:30:12.288890 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.233624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60635 (local address 127.0.97.60:34305)
0620 18:30:12.269022 (+ 35398us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.269031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.269055 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:30:12.269126 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.269134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.269228 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.269349 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.272186 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.273327 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.276261 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.276894 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.276996 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.281339 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.281347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.281352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.281394 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.284978 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.285744 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.285751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.285754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.285940 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286382 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.286386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.286390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.286716 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.286855 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.288277 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.288489 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35232,"thread_start_us":35077,"threads_started":1}
I20260620 18:30:12.288975 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.237683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.0.97.3:44153)
0620 18:30:12.257224 (+ 19541us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.257237 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.258133 (+ 896us) server_negotiation.cc:407] Connection header received
0620 18:30:12.258239 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.258251 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.258573 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.258751 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.266523 (+ 7772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.268451 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.271843 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.272821 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.272945 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.280111 (+ 7166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.280127 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.280144 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.280191 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.284766 (+ 4575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.285556 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.285564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.285567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.285720 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286176 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.286181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.286184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.286550 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.286827 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.287956 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.288213 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19326,"thread_start_us":19082,"threads_started":1}
I20260620 18:30:12.289667 7658 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35353
I20260620 18:30:12.289999 7658 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.290230 6994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:12.291007 7658 heartbeater.cc:511] Master 127.0.97.61:35353 requested a full tablet report, sending...
I20260620 18:30:12.292312 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34305
I20260620 18:30:12.292477 7656 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.292882 7656 heartbeater.cc:511] Master 127.0.97.60:34305 requested a full tablet report, sending...
I20260620 18:30:12.293455 7142 ts_manager.cc:194] Registered new tserver with Master: 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:12.294961 7332 ts_manager.cc:194] Registered new tserver with Master: 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:12.295482 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.234523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:48391)
0620 18:30:12.270009 (+ 35486us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.270030 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.270133 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.270869 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.270882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.271614 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.272078 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.272091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.274830 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.274841 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.276482 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.276495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.277609 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.277616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.277787 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.278355 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.278378 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.281300 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.285226 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.285230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.285234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.285987 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.286396 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.286401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.286404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.286529 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.287159 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.287165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.287277 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.287411 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.287679 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35367,"mutex_wait_us":66,"thread_start_us":24634,"threads_started":1}
I20260620 18:30:12.296550 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:12.296686 7657 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.297091 7657 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
I20260620 18:30:12.298323 6543 ts_manager.cc:194] Re-registered known tserver with Master: 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:12.299685 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48391
I20260620 18:30:12.377635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6871
I20260620 18:30:12.393858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44153
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45127
--webserver_port=34689
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:30:12.670261 7511 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
W20260620 18:30:12.716702 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:12.716975 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:12.717067 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:12.724334 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:12.724484 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:12.724658 7669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:12.734864 7669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34689
--tserver_master_addrs=127.0.97.62:45127,127.0.97.61:35353,127.0.97.60:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:12.736218 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:12.737437 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:12.750491 7675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:12.750552 7678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:12.750514 7676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:12.751473 7669 server_base.cc:1034] running on GCE node
I20260620 18:30:12.752185 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:12.753151 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:12.754385 7669 hybrid_clock.cc:648] HybridClock initialized: now 1781980212754339 us; error 39 us; skew 500 ppm
I20260620 18:30:12.754701 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:12.757858 7669 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 18:30:12.759618 7669 webserver.cc:460] Webserver started at http://127.0.97.3:34689/ using document root <none> and password file <none>
I20260620 18:30:12.760319 7669 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:12.760488 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:12.768605 7669 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:30:12.771440 7685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.773104 7669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:30:12.773320 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "41a08f8b79534940a66de768c7ca02a5"
format_stamp: "Formatted at 2026-06-20 18:30:09 on dist-test-slave-mcnm"
server_key: "7fba4a160a28b5b9a1d53963c7026aa8"
server_key_iv: "db7d1f524b72269edb77f015aabe5f6d"
server_key_version: "encryptionkey@0"
I20260620 18:30:12.774434 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:12.800009 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:12.804276 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:12.804576 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:12.805573 7669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:12.807112 7669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:12.807260 7669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.807392 7669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:12.807488 7669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:12.844046 7669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44153
I20260620 18:30:12.844134 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44153 every 8 connection(s)
I20260620 18:30:12.845569 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:12.855340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7669
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 18:30:12 dist-test-slave-mcnm krb5kdc[6498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:12.885231 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.849378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.97.3:43303)
0620 18:30:12.850260 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.850324 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.853918 (+ 3594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.861337 (+ 7419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.861347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.862135 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.863734 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.863759 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.865049 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.865054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.867533 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.867553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.869313 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.869324 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.869579 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.870289 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.870320 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.874228 (+ 3908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.879363 (+ 5135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.879376 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.879401 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.880110 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.880585 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.880589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.880593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.880885 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.881762 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.881774 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.882336 (+ 562us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.882684 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.883407 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":896,"thread_start_us":2378,"threads_started":1}
I20260620 18:30:12.886878 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.848830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43303 (local address 127.0.97.61:35353)
0620 18:30:12.849263 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.849267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.850503 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 18:30:12.854334 (+ 3831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.854344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.854478 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.854577 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.863892 (+ 9315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.864939 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.867743 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.868634 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.868756 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.874555 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.874564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.874571 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.874639 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.879125 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.880284 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.880289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.880292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.880458 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.881049 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.881057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.881063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.881500 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.881676 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.886046 (+ 4370us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.886363 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
I20260620 18:30:12.891942 7800 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35353
I20260620 18:30:12.892293 7800 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.893249 7800 heartbeater.cc:511] Master 127.0.97.61:35353 requested a full tablet report, sending...
I20260620 18:30:12.895051 7142 ts_manager.cc:194] Registered new tserver with Master: 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153)
I20260620 18:30:12.895350 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.850721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55661 (local address 127.0.97.62:45127)
0620 18:30:12.862628 (+ 11907us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.862634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.863052 (+ 418us) server_negotiation.cc:407] Connection header received
0620 18:30:12.863123 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.863128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.863222 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.863344 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.864445 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.865370 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.868476 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.868869 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.869009 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.882527 (+ 13518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.882537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.882544 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.882596 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.886872 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.887939 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.887948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.887954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.888128 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.893799 (+ 5671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.893806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.893812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.894138 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.894292 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.894787 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.894997 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11780,"thread_start_us":11904,"threads_started":1}
I20260620 18:30:12.896056 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.850020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.97.3:56097)
0620 18:30:12.856471 (+ 6451us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.856497 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.856647 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.861833 (+ 5186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.861838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.862506 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.863249 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.863285 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.864993 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.864998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.866653 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.866665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.867887 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.867893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.868191 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.870175 (+ 1984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.870194 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.888520 (+ 18326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.893079 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.893086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.893096 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.893621 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.894047 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.894051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.894055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.894194 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.894908 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.894915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.895069 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.895251 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.895582 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":6,"thread_start_us":836,"threads_started":1}
I20260620 18:30:12.896708 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.850655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:55661)
0620 18:30:12.862716 (+ 12061us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.862736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.862864 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.863364 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.863369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.864001 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:12.864261 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.864268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.866847 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.866855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.868343 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.868354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.869228 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.869246 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.869472 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.870075 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:12.870108 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:12.882236 (+ 12128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:12.887075 (+ 4839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.887083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.887089 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.887774 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.888283 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:12.888290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:12.888296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:12.888544 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:12.894372 (+ 5828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:12.894378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:12.894514 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.895812 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.896307 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11946,"mutex_wait_us":54,"spinlock_wait_cycles":24832,"thread_start_us":12235,"threads_started":1}
I20260620 18:30:12.896943 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.850057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56097 (local address 127.0.97.60:34305)
0620 18:30:12.861411 (+ 11354us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.861420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.861445 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:30:12.861517 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.861526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.861641 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.861784 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.863561 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.864824 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.866810 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.867445 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.867592 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.888764 (+ 21172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.888775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.888782 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.888832 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.892921 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.893754 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.893762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.893766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.893910 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.894346 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.894352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.894358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.894699 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.894834 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.896349 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.896585 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11190,"thread_start_us":11062,"threads_started":1}
I20260620 18:30:12.897712 7799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45127
I20260620 18:30:12.897841 7799 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.898286 7799 heartbeater.cc:511] Master 127.0.97.62:45127 requested a full tablet report, sending...
I20260620 18:30:12.899024 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34305
I20260620 18:30:12.899147 7798 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:12.899540 7798 heartbeater.cc:511] Master 127.0.97.60:34305 requested a full tablet report, sending...
I20260620 18:30:12.899962 6543 ts_manager.cc:194] Re-registered known tserver with Master: 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153)
I20260620 18:30:12.900620 7332 ts_manager.cc:194] Registered new tserver with Master: 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153)
I20260620 18:30:12.901052 6543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55661
I20260620 18:30:12.912328 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.872780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.97.60:34305)
0620 18:30:12.880704 (+ 7924us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.880713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.890370 (+ 9657us) server_negotiation.cc:407] Connection header received
0620 18:30:12.890548 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.890555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.890666 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.890785 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.898174 (+ 7389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.899317 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.902058 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.902702 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.902819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.905897 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.905903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.905906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.905933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.909111 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.909841 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.909846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.909850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.910015 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.910468 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.910473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.910476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.910784 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.911050 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.911688 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.911928 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":251,"threads_started":1}
I20260620 18:30:12.913080 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.873494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.0.97.62:45127)
0620 18:30:12.893103 (+ 19609us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.893109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.893126 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:12.893190 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.893197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.893268 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.893365 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.896614 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.898207 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.901533 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.902369 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.902505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.906600 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.906606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.906611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.906651 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.909886 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.910612 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.910619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.910623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.910776 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.911166 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.911171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.911174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.911505 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.911768 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.912469 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.912681 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19495,"thread_start_us":23483,"threads_started":1}
I20260620 18:30:12.913918 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.873134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.97.61:35353)
0620 18:30:12.887115 (+ 13981us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.887120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.889695 (+ 2575us) server_negotiation.cc:407] Connection header received
0620 18:30:12.890994 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.891001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.891085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.891158 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:12.893249 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.894301 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.900442 (+ 6141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.901061 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.901161 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.907289 (+ 6128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:12.907296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:12.907301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:12.907347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:12.910846 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.911469 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.911475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.911478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.911648 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:12.912064 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:12.912070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:12.912073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:12.912410 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:12.912793 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.913370 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.913588 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13882,"thread_start_us":17516,"threads_started":1}
I20260620 18:30:12.917552 6543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:12.925218 6543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:12.957813 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.948721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.97.1:37701)
0620 18:30:12.949245 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.949255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.949850 (+ 595us) server_negotiation.cc:407] Connection header received
0620 18:30:12.950085 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.950092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.950297 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.950504 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:12.951029 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.952478 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.955149 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.956675 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.956826 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.957041 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.957272 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.957454 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260620 18:30:12.958248 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.948473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37701 (local address 127.0.0.1:45216)
0620 18:30:12.949732 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.949748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.949902 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.950476 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.950481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.950505 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:12.950858 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.950866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.952626 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.952634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.954944 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.954958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.956886 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.956892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.957062 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.957097 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.957763 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.957907 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
I20260620 18:30:12.959797 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.949053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33521 (local address 127.0.0.1:48296)
0620 18:30:12.952060 (+ 3007us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.952077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.952302 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.953278 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.953285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.953319 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:12.953785 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.953799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.955451 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.955457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.957365 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.957376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.959039 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.959045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.959214 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.959248 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.959340 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.959457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2877,"thread_start_us":2701,"threads_started":1}
I20260620 18:30:12.961722 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 (DEFAULT_TABLE table=test-table [id=7b3855578ba0404bac28283406b1a8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:12.963150 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd1d22c42c487e8b62ebe186c169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:12.963681 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.952070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44153 (local address 127.0.0.1:39428)
0620 18:30:12.954832 (+ 2762us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:12.954848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:12.954952 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:12.957129 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:12.957136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:12.957167 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:12.957600 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.957613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.960255 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.960262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.961484 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:12.961495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.962918 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.962924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:12.963120 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.963148 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:30:12.963232 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:30:12.963347 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2630,"thread_start_us":2504,"threads_started":1}
I20260620 18:30:12.963907 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.952449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.0.97.3:44153)
0620 18:30:12.953176 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.953184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.955494 (+ 2310us) server_negotiation.cc:407] Connection header received
0620 18:30:12.955619 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.955628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.955880 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.956064 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:12.958609 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.960116 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.961636 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.962700 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.962845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.963045 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.963413 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.963603 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":387,"threads_started":1}
I20260620 18:30:12.966650 7733 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 (DEFAULT_TABLE table=test-table [id=7b3855578ba0404bac28283406b1a8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:12.967777 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:12.949281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.97.2:33521)
0620 18:30:12.949860 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:12.949873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:12.952168 (+ 2295us) server_negotiation.cc:407] Connection header received
0620 18:30:12.952495 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:12.952504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:12.952802 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:12.952973 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:12.953972 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.955345 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.957500 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:12.958824 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:12.958943 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:12.959132 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:12.967289 (+ 8157us) server_negotiation.cc:299] Negotiation successful
0620 18:30:12.967485 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260620 18:30:12.967864 7733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd1d22c42c487e8b62ebe186c169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:12.971062 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 (DEFAULT_TABLE table=test-table [id=7b3855578ba0404bac28283406b1a8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:12.972879 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd1d22c42c487e8b62ebe186c169f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:12.983712 7832 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Bootstrap starting.
I20260620 18:30:12.987179 7832 tablet_bootstrap.cc:654] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:12.990067 7832 log.cc:824] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:12.991981 7833 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Bootstrap starting.
I20260620 18:30:12.996107 7834 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Bootstrap starting.
I20260620 18:30:12.996417 7833 tablet_bootstrap.cc:654] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:12.997644 7833 log.cc:824] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:12.999938 7832 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: No bootstrap required, opened a new log
I20260620 18:30:13.000129 7834 tablet_bootstrap.cc:654] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:13.000392 7832 ts_tablet_manager.cc:1397] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260620 18:30:13.001801 7833 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: No bootstrap required, opened a new log
I20260620 18:30:13.001899 7834 log.cc:824] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:13.002090 7833 ts_tablet_manager.cc:1397] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:30:13.006531 7834 tablet_bootstrap.cc:492] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: No bootstrap required, opened a new log
I20260620 18:30:13.006804 7834 ts_tablet_manager.cc:1397] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 18:30:13.009428 7832 raft_consensus.cc:348] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.009927 7832 raft_consensus.cc:374] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:13.010027 7832 raft_consensus.cc:729] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47b6ef6f62f4aa0bc3721b1b3c3a3b1, State: Initialized, Role: FOLLOWER
I20260620 18:30:13.010627 7832 consensus_queue.cc:260] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.012876 7833 raft_consensus.cc:348] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.013338 7833 raft_consensus.cc:374] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:13.013437 7833 raft_consensus.cc:729] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a08f8b79534940a66de768c7ca02a5, State: Initialized, Role: FOLLOWER
I20260620 18:30:13.014038 7833 consensus_queue.cc:260] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.016957 7833 ts_tablet_manager.cc:1428] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 18:30:13.017402 7799 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
I20260620 18:30:13.017791 7834 raft_consensus.cc:348] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.018201 7834 raft_consensus.cc:374] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:13.018730 7834 raft_consensus.cc:729] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0ae4d34fac454995aa3508ec23357e, State: Initialized, Role: FOLLOWER
I20260620 18:30:13.019454 7834 consensus_queue.cc:260] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.021386 7832 ts_tablet_manager.cc:1428] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
I20260620 18:30:13.027479 7657 heartbeater.cc:503] Master 127.0.97.62:45127 was elected leader, sending a full tablet report...
I20260620 18:30:13.028054 7834 ts_tablet_manager.cc:1428] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Time spent starting tablet: real 0.021s user 0.016s sys 0.001s
I20260620 18:30:13.054055 7839 raft_consensus.cc:482] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:13.054370 7839 raft_consensus.cc:504] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.056106 7839 leader_election.cc:288] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701), 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:13.061872 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.056515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50703 (local address 127.0.97.1:37701)
0620 18:30:13.056783 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.056790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.056811 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:30:13.057024 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.057032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.057106 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.057225 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:13.057652 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.058405 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.060159 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.060906 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.060998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.061092 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.061504 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.061623 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:30:13.061916 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.056936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59765 (local address 127.0.97.2:33521)
0620 18:30:13.057117 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.057122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.057136 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:30:13.057233 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.057238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.057308 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.057401 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:13.057778 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.058433 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.060202 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.061006 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.061126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.061234 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.061552 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.061683 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:30:13.061913 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.056764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33521 (local address 127.0.97.3:59765)
0620 18:30:13.056989 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:13.057005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:13.057102 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:13.057429 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:13.057432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:13.057454 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:13.057635 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:13.057645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.058602 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.058605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:13.060026 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:13.060037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.061153 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.061157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:13.061366 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.061388 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:30:13.061493 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:30:13.061631 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":83}
I20260620 18:30:13.061916 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.056424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37701 (local address 127.0.97.3:50703)
0620 18:30:13.056715 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:13.056738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:13.056864 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:13.057245 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:13.057250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:13.057275 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:13.057490 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:13.057503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.058554 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.058559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:13.060010 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:13.060021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.061058 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.061068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:13.061339 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.061374 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:30:13.061443 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:30:13.061602 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":67}
I20260620 18:30:13.062964 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd1d22c42c487e8b62ebe186c169f2" candidate_uuid: "41a08f8b79534940a66de768c7ca02a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f0ae4d34fac454995aa3508ec23357e" is_pre_election: true
I20260620 18:30:13.062966 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd1d22c42c487e8b62ebe186c169f2" candidate_uuid: "41a08f8b79534940a66de768c7ca02a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" is_pre_election: true
I20260620 18:30:13.063367 7611 raft_consensus.cc:2454] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a08f8b79534940a66de768c7ca02a5 in term 0.
I20260620 18:30:13.064033 7465 raft_consensus.cc:2454] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a08f8b79534940a66de768c7ca02a5 in term 0.
I20260620 18:30:13.064178 7689 leader_election.cc:302] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0ae4d34fac454995aa3508ec23357e, 41a08f8b79534940a66de768c7ca02a5; no voters:
I20260620 18:30:13.064616 7839 raft_consensus.cc:2781] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:13.064775 7839 raft_consensus.cc:482] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:13.064883 7839 raft_consensus.cc:3037] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:13.066982 7839 raft_consensus.cc:504] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.067566 7839 leader_election.cc:288] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [CANDIDATE]: Term 1 election: Requested vote from peers a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701), 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521)
I20260620 18:30:13.067965 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd1d22c42c487e8b62ebe186c169f2" candidate_uuid: "41a08f8b79534940a66de768c7ca02a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
I20260620 18:30:13.067986 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd1d22c42c487e8b62ebe186c169f2" candidate_uuid: "41a08f8b79534940a66de768c7ca02a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f0ae4d34fac454995aa3508ec23357e"
I20260620 18:30:13.068164 7465 raft_consensus.cc:3037] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:13.068176 7611 raft_consensus.cc:3037] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:13.070597 7465 raft_consensus.cc:2454] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a08f8b79534940a66de768c7ca02a5 in term 1.
I20260620 18:30:13.070611 7611 raft_consensus.cc:2454] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a08f8b79534940a66de768c7ca02a5 in term 1.
I20260620 18:30:13.070966 7689 leader_election.cc:302] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a08f8b79534940a66de768c7ca02a5, a47b6ef6f62f4aa0bc3721b1b3c3a3b1; no voters:
I20260620 18:30:13.071277 7839 raft_consensus.cc:2781] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:13.071750 7839 raft_consensus.cc:686] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 1 LEADER]: Becoming Leader. State: Replica: 41a08f8b79534940a66de768c7ca02a5, State: Running, Role: LEADER
I20260620 18:30:13.072489 7839 consensus_queue.cc:237] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } }
I20260620 18:30:13.077466 6542 catalog_manager.cc:5495] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "41a08f8b79534940a66de768c7ca02a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } health_report { overall_health: HEALTHY } } }
W20260620 18:30:13.098253 7801 tablet.cc:2369] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:13.110729 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.105599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.0.97.3:44153)
0620 18:30:13.105802 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.105808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.105826 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:13.106056 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.106060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.106134 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.106279 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:13.106625 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.107390 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.108720 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.109104 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.109230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.109682 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:13.110157 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:13.110316 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.110415 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.110490 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:30:13.118436 7465 raft_consensus.cc:1261] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Refusing update from remote peer 41a08f8b79534940a66de768c7ca02a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:13.118438 7611 raft_consensus.cc:1261] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Refusing update from remote peer 41a08f8b79534940a66de768c7ca02a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:13.119448 7841 consensus_queue.cc:1035] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:13.119961 7839 consensus_queue.cc:1035] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:30:13.128124 7513 tablet.cc:2369] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:13.137333 7844 mvcc.cc:205] Tried to move back new op lower bound from 7298990952923693056 to 7298990952749051904. Current Snapshot: MvccSnapshot[applied={T|T < 7298990952923693056}]
I20260620 18:30:13.160655 6542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:13.160957 6542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:13.167392 6542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 340bd78a35364e1cbc026f88541efd29: Sending DeleteTablet for 3 replicas of tablet 24bd1d22c42c487e8b62ebe186c169f2
I20260620 18:30:13.168687 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45216
I20260620 18:30:13.169678 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48296
I20260620 18:30:13.170469 7733 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bd1d22c42c487e8b62ebe186c169f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39428
I20260620 18:30:13.184038 7854 tablet_replica.cc:330] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: stopping tablet replica
I20260620 18:30:13.184584 7854 raft_consensus.cc:2229] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:13.185153 7854 raft_consensus.cc:2258] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:13.187582 7854 ts_tablet_manager.cc:1905] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:13.194444 7857 tablet_replica.cc:330] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: stopping tablet replica
I20260620 18:30:13.194979 7857 raft_consensus.cc:2229] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:13.195740 7854 ts_tablet_manager.cc:1918] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:13.195902 7854 log.cc:1196] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/24bd1d22c42c487e8b62ebe186c169f2
I20260620 18:30:13.196441 7854 ts_tablet_manager.cc:1939] T 24bd1d22c42c487e8b62ebe186c169f2 P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting consensus metadata
I20260620 18:30:13.197167 7855 tablet_replica.cc:330] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: stopping tablet replica
I20260620 18:30:13.197300 7857 raft_consensus.cc:2258] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:13.197666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 6511
I20260620 18:30:13.198055 7855 raft_consensus.cc:2229] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:13.198408 6530 catalog_manager.cc:4841] TS a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701): tablet 24bd1d22c42c487e8b62ebe186c169f2 (table test-table [id=7b3855578ba0404bac28283406b1a8c9]) successfully deleted
I20260620 18:30:13.198642 7855 raft_consensus.cc:2258] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:13.198681 6573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:13.200646 7857 ts_tablet_manager.cc:1905] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:13.200896 7855 ts_tablet_manager.cc:1905] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:13.207799 7855 ts_tablet_manager.cc:1918] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:13.208039 7855 log.cc:1196] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/24bd1d22c42c487e8b62ebe186c169f2
I20260620 18:30:13.208140 7857 ts_tablet_manager.cc:1918] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:13.208303 7857 log.cc:1196] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/24bd1d22c42c487e8b62ebe186c169f2
I20260620 18:30:13.208949 7857 ts_tablet_manager.cc:1939] T 24bd1d22c42c487e8b62ebe186c169f2 P 41a08f8b79534940a66de768c7ca02a5: Deleting consensus metadata
I20260620 18:30:13.219944 7855 ts_tablet_manager.cc:1939] T 24bd1d22c42c487e8b62ebe186c169f2 P 3f0ae4d34fac454995aa3508ec23357e: Deleting consensus metadata
I20260620 18:30:13.223474 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.171071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.0.97.61:35353)
0620 18:30:13.171230 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.171238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.195923 (+ 24685us) server_negotiation.cc:407] Connection header received
0620 18:30:13.196753 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.196759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.196843 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.196939 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:13.209456 (+ 12517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.211030 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.212948 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.213597 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.213686 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.217081 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:13.217087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:13.217092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:13.217130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:13.220604 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.221440 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.221445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.221448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.221587 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.222026 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.222031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.222034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.222308 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:13.222581 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.222922 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.223122 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:30:13.248343 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.195819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.97.60:34305)
0620 18:30:13.196090 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.196095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.223681 (+ 27586us) server_negotiation.cc:407] Connection header received
0620 18:30:13.224115 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.224122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.224202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.224267 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:13.226296 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.226973 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.236202 (+ 9229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.236874 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.236968 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.240323 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:13.240330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:13.240335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:13.240374 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:13.244886 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.246109 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.246116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.246120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.246262 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.246654 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.246659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.246662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.247004 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:13.247224 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.247829 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.248048 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 18:30:13.398782 6530 catalog_manager.cc:4841] TS 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521): tablet 24bd1d22c42c487e8b62ebe186c169f2 (table test-table [id=7b3855578ba0404bac28283406b1a8c9]) successfully deleted
I20260620 18:30:13.399165 6530 catalog_manager.cc:4841] TS 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153): tablet 24bd1d22c42c487e8b62ebe186c169f2 (table test-table [id=7b3855578ba0404bac28283406b1a8c9]) successfully deleted
I20260620 18:30:13.399596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 6511
I20260620 18:30:13.449299 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.425929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.0.97.61:35353)
0620 18:30:13.426146 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.426155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.432555 (+ 6400us) server_negotiation.cc:407] Connection header received
0620 18:30:13.432852 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.432857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.432935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.433006 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:13.435639 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.436741 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.438562 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.439127 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.439274 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.442941 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:13.442950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:13.442955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:13.442998 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:13.446394 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.447300 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.447307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.447313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.447438 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.447865 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.447874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.447880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.448163 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:13.448463 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.448779 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.448979 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:30:13.450011 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:13.425503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.0.97.60:34305)
0620 18:30:13.425684 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:13.425691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:13.432555 (+ 6864us) server_negotiation.cc:407] Connection header received
0620 18:30:13.432855 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:13.432860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:13.432935 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:13.433006 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:13.435826 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.436538 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.438537 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:13.439255 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:13.439378 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:13.443711 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:13.443719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:13.443725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:13.443807 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:13.446431 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.447220 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.447226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.447230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.447400 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:13.447925 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:13.447930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:13.447933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:13.448308 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:13.448607 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:13.449325 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 18:30:13.449695 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:30:14.143878 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.143039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:53249)
0620 18:30:14.143611 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.143726 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
W20260620 18:30:14.144398 7511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.146560 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.145859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:60549)
0620 18:30:14.146269 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.146387 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":205,"threads_started":1}
W20260620 18:30:14.147084 7799 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.150399 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.149638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:55607)
0620 18:30:14.150070 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.150215 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":252,"threads_started":1}
W20260620 18:30:14.150789 7657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.667851 7869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 340bd78a35364e1cbc026f88541efd29)
I20260620 18:30:14.668188 7869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:14.668335 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 340bd78a35364e1cbc026f88541efd29)
I20260620 18:30:14.668610 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:14.669188 7869 leader_election.cc:288] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 1774525a63a343869db365001e3b30fd (127.0.97.60:34305)
I20260620 18:30:14.670288 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61:35353)
I20260620 18:30:14.671391 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.669360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38960)
0620 18:30:14.670856 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.671061 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1343,"threads_started":1}
I20260620 18:30:14.673127 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.670314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38976)
0620 18:30:14.672504 (+ 2190us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.672723 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2066,"thread_start_us":1940,"threads_started":1}
W20260620 18:30:14.673389 7317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
W20260620 18:30:14.673856 7128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.676102 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.675445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:39002)
0620 18:30:14.675699 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.675866 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 18:30:14.676352 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.676946 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.674709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:38978)
0620 18:30:14.675242 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.676609 (+ 1367us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":1304,"threads_started":1}
W20260620 18:30:14.677105 7317 leader_election.cc:334] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.681118 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.673388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35353 (local address 127.0.0.1:44738)
0620 18:30:14.673652 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.673669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:14.673774 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:14.675661 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:14.675667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:14.675699 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:14.675938 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:14.675951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.677556 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.677562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:14.679075 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:14.679086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.680454 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.680460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:14.680630 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:14.680672 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:30:14.680787 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:30:14.680923 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 18:30:14.681353 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.673046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.0.97.61:35353)
0620 18:30:14.674157 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:14.674163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:14.674180 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:14.674232 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:14.674243 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:14.674312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:14.676094 (+ 1782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:14.676357 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.677423 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.679243 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.680279 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.680381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:14.680446 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:14.680838 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 18:30:14.680980 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":769,"threads_started":1}
I20260620 18:30:14.681746 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.669510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34305 (local address 127.0.0.1:33386)
0620 18:30:14.671693 (+ 2183us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.671717 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:14.671938 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:14.674029 (+ 2091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:14.674035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:14.674063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:14.674322 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:14.674333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.677984 (+ 3651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.677987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:14.679655 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:14.679668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.680908 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.680914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:14.681094 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:14.681134 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:30:14.681222 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:30:14.681359 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"thread_start_us":1067,"threads_started":1}
I20260620 18:30:14.681911 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.669679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.97.60:34305)
0620 18:30:14.672793 (+ 3114us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:14.672799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:14.672866 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:30:14.672934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:14.672939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:14.673059 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:14.676377 (+ 3318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:14.676561 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.677847 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.679798 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:14.680720 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:14.680836 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:14.680931 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:14.681469 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 18:30:14.681597 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2989,"thread_start_us":2852,"threads_started":1}
I20260620 18:30:14.681977 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1774525a63a343869db365001e3b30fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f59276d8d74d2f891944e602f2c4ac" is_pre_election: true
I20260620 18:30:14.682305 7152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1774525a63a343869db365001e3b30fd in term 1.
I20260620 18:30:14.682598 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f59276d8d74d2f891944e602f2c4ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1774525a63a343869db365001e3b30fd" is_pre_election: true
I20260620 18:30:14.682806 7315 leader_election.cc:302] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1774525a63a343869db365001e3b30fd, 39f59276d8d74d2f891944e602f2c4ac; no voters: 340bd78a35364e1cbc026f88541efd29
I20260620 18:30:14.682942 7342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f59276d8d74d2f891944e602f2c4ac in term 1.
I20260620 18:30:14.683187 7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:30:14.683308 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 340bd78a35364e1cbc026f88541efd29)
I20260620 18:30:14.683418 7870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:30:14.683609 7129 leader_election.cc:302] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1774525a63a343869db365001e3b30fd, 39f59276d8d74d2f891944e602f2c4ac; no voters: 340bd78a35364e1cbc026f88541efd29
I20260620 18:30:14.683914 7869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:30:14.684062 7869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 340bd78a35364e1cbc026f88541efd29)
I20260620 18:30:14.684175 7869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:30:14.685665 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:14.686303 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 election: Requested vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 39f59276d8d74d2f891944e602f2c4ac (127.0.97.61:35353)
I20260620 18:30:14.686230 7869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:14.686772 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.686408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:39018)
0620 18:30:14.686539 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.686622 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:30:14.686708 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1774525a63a343869db365001e3b30fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f59276d8d74d2f891944e602f2c4ac"
I20260620 18:30:14.686902 7869 leader_election.cc:288] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 election: Requested vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 1774525a63a343869db365001e3b30fd (127.0.97.60:34305)
I20260620 18:30:14.687000 7152 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1774525a63a343869db365001e3b30fd in current term 2: Already voted for candidate 39f59276d8d74d2f891944e602f2c4ac in this term.
I20260620 18:30:14.687314 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.686927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:39030)
0620 18:30:14.687082 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.687169 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:30:14.687629 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f59276d8d74d2f891944e602f2c4ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1774525a63a343869db365001e3b30fd"
I20260620 18:30:14.687906 7342 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f59276d8d74d2f891944e602f2c4ac in current term 2: Already voted for candidate 1774525a63a343869db365001e3b30fd in this term.
I20260620 18:30:14.687945 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.687564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:39034)
0620 18:30:14.687709 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.687811 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:30:14.688416 7317 leader_election.cc:334] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.688470 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:14.688183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:39038)
0620 18:30:14.688310 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:14.688367 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:30:14.688567 7317 leader_election.cc:302] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1774525a63a343869db365001e3b30fd; no voters: 340bd78a35364e1cbc026f88541efd29, 39f59276d8d74d2f891944e602f2c4ac
I20260620 18:30:14.688858 7870 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 18:30:14.689002 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:14.689134 7128 leader_election.cc:302] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f59276d8d74d2f891944e602f2c4ac; no voters: 1774525a63a343869db365001e3b30fd, 340bd78a35364e1cbc026f88541efd29
I20260620 18:30:14.689391 7869 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:30:15.145862 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:15.145113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:33583)
0620 18:30:15.145595 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:15.145677 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":275,"threads_started":1}
I20260620 18:30:15.148651 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:15.147794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:44131)
0620 18:30:15.148259 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:15.148360 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":216,"threads_started":1}
I20260620 18:30:15.151933 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:15.151320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:46049)
0620 18:30:15.151718 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:15.151785 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":222,"threads_started":1}
I20260620 18:30:16.147698 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.146786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:60451)
0620 18:30:16.147384 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.147480 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":404,"threads_started":1}
I20260620 18:30:16.150403 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.149632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:57153)
0620 18:30:16.150110 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.150210 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":282,"threads_started":1}
I20260620 18:30:16.153167 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.152603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:40365)
0620 18:30:16.152951 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.153030 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":174,"threads_started":1}
I20260620 18:30:16.488543 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:16.488750 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:16.489411 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 1774525a63a343869db365001e3b30fd (127.0.97.60:34305)
I20260620 18:30:16.489959 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f59276d8d74d2f891944e602f2c4ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1774525a63a343869db365001e3b30fd" is_pre_election: true
I20260620 18:30:16.490206 7342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f59276d8d74d2f891944e602f2c4ac in term 2.
I20260620 18:30:16.490500 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.489659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55596)
0620 18:30:16.490175 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.490260 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
I20260620 18:30:16.490659 7129 leader_election.cc:302] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1774525a63a343869db365001e3b30fd, 39f59276d8d74d2f891944e602f2c4ac; no voters:
I20260620 18:30:16.490996 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 18:30:16.491147 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:16.491266 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:30:16.492000 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.491656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55610)
0620 18:30:16.491808 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.491892 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:30:16.492169 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:16.493554 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:16.494074 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 election: Requested vote from peers 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127), 1774525a63a343869db365001e3b30fd (127.0.97.60:34305)
I20260620 18:30:16.494593 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.494243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55624)
0620 18:30:16.494369 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.494467 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:30:16.494807 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f59276d8d74d2f891944e602f2c4ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1774525a63a343869db365001e3b30fd"
I20260620 18:30:16.495131 7342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:30:16.495772 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.495425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55640)
0620 18:30:16.495607 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.495672 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:30:16.495994 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
I20260620 18:30:16.497712 7342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f59276d8d74d2f891944e602f2c4ac in term 3.
I20260620 18:30:16.498153 7129 leader_election.cc:302] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1774525a63a343869db365001e3b30fd, 39f59276d8d74d2f891944e602f2c4ac; no voters: 340bd78a35364e1cbc026f88541efd29
I20260620 18:30:16.498481 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:30:16.498977 7885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [term 3 LEADER]: Becoming Leader. State: Replica: 39f59276d8d74d2f891944e602f2c4ac, State: Running, Role: LEADER
I20260620 18:30:16.499351 7885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } }
I20260620 18:30:16.501616 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39f59276d8d74d2f891944e602f2c4ac. Latest consensus state: current_term: 3 leader_uuid: "39f59276d8d74d2f891944e602f2c4ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:16.501848 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:16.502444 7890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:16.505812 7890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:16.506577 7890 catalog_manager.cc:1259] Loaded cluster ID: e6b6a77882c94f9b8017a00e1c872585
I20260620 18:30:16.506644 7890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:16.507799 7890 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:16.508551 7890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Loaded TSK: 0
I20260620 18:30:16.509387 7890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:30:16.559587 7142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:16.566453 7142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:16.570607 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.570126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55652)
0620 18:30:16.570303 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.570437 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:30:16.570715 7342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [term 3 FOLLOWER]: Refusing update from remote peer 39f59276d8d74d2f891944e602f2c4ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 18:30:16.571446 7888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:30:16.571702 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.571353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55658)
0620 18:30:16.571489 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.571605 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 18:30:16.571973 7128 consensus_peers.cc:487] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac -> Peer 340bd78a35364e1cbc026f88541efd29 (127.0.97.62:45127): Couldn't send request to peer 340bd78a35364e1cbc026f88541efd29. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:30:16.574451 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39f59276d8d74d2f891944e602f2c4ac. Latest consensus state: current_term: 3 leader_uuid: "39f59276d8d74d2f891944e602f2c4ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:16.574720 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:30:16.575984 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "39f59276d8d74d2f891944e602f2c4ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:16.576359 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:16.577245 7893 mvcc.cc:205] Tried to move back new op lower bound from 7298990967066599424 to 7298990966784720896. Current Snapshot: MvccSnapshot[applied={T|T < 7298990967066599424}]
I20260620 18:30:16.577152 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "39f59276d8d74d2f891944e602f2c4ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "340bd78a35364e1cbc026f88541efd29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45127 } } peers { permanent_uuid: "39f59276d8d74d2f891944e602f2c4ac" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35353 } attrs { promote: false } } peers { permanent_uuid: "1774525a63a343869db365001e3b30fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34305 } attrs { promote: false } } }
I20260620 18:30:16.577451 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1774525a63a343869db365001e3b30fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:30:16.594420 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.584351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44153 (local address 127.0.0.1:38690)
0620 18:30:16.584531 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.584551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:16.584662 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:16.586078 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:16.586085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:16.586115 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:16.586442 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.586456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.588491 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.588499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.591171 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.591191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.593404 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.593413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.593668 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.593724 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:30:16.593804 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:30:16.594005 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:30:16.595729 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.584443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.0.97.3:44153)
0620 18:30:16.585690 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.585696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.585715 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:30:16.585780 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.585785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.585867 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.586606 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:16.586882 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.588286 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.591428 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.593179 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.594828 (+ 1649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.594997 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.595062 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.595255 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":989,"threads_started":1}
I20260620 18:30:16.597525 7733 tablet_service.cc:1467] Processing CreateTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e (DEFAULT_TABLE table=test-table [id=cb270b054c754a8f81943ba0eb787f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:16.597946 7733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907113bdcee40d8bb5e7d4d95ce307e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:16.601275 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.585145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.97.2:33521)
0620 18:30:16.595500 (+ 10355us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.595504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.595571 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:30:16.595638 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.595642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.595695 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.595802 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:16.596418 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.597120 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.599085 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.600131 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.600256 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.600367 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.600829 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.600961 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10240,"thread_start_us":10129,"threads_started":1}
I20260620 18:30:16.601315 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.584669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33521 (local address 127.0.0.1:52420)
0620 18:30:16.585248 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.585268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:16.585374 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:16.595814 (+ 10440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:16.595819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:16.595844 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:16.596202 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.596213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.597242 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.597246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.598902 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.598913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.600507 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.600515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.600676 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.600700 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:30:16.600804 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:30:16.600936 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1,"thread_start_us":630,"threads_started":1}
I20260620 18:30:16.602321 7591 tablet_service.cc:1467] Processing CreateTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e (DEFAULT_TABLE table=test-table [id=cb270b054c754a8f81943ba0eb787f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:16.602721 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.585626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37701 (local address 127.0.0.1:33154)
0620 18:30:16.586169 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.586194 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:16.586330 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:16.596632 (+ 10302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:16.596637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:16.596661 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:16.596915 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.596922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.598305 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.598311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.600424 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.600441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.601990 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.601998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.602167 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.602198 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:30:16.602290 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:30:16.602416 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":278,"threads_started":1}
I20260620 18:30:16.602766 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907113bdcee40d8bb5e7d4d95ce307e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:16.602775 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.585765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.97.1:37701)
0620 18:30:16.596364 (+ 10599us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.596371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.596388 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:16.596452 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.596456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.596515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.596621 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:16.597057 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.598170 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.600611 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.601769 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.601899 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.602042 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.602329 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.602465 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10452,"thread_start_us":10346,"threads_started":1}
I20260620 18:30:16.603432 7445 tablet_service.cc:1467] Processing CreateTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e (DEFAULT_TABLE table=test-table [id=cb270b054c754a8f81943ba0eb787f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:16.603863 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907113bdcee40d8bb5e7d4d95ce307e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:16.604637 7903 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Bootstrap starting.
I20260620 18:30:16.606463 7903 tablet_bootstrap.cc:654] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:16.608712 7903 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: No bootstrap required, opened a new log
I20260620 18:30:16.608906 7903 ts_tablet_manager.cc:1397] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:30:16.609721 7905 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Bootstrap starting.
I20260620 18:30:16.609777 7903 raft_consensus.cc:348] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.610126 7903 raft_consensus.cc:374] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:16.610287 7903 raft_consensus.cc:729] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a08f8b79534940a66de768c7ca02a5, State: Initialized, Role: FOLLOWER
I20260620 18:30:16.610576 7903 consensus_queue.cc:260] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.611727 7903 ts_tablet_manager.cc:1428] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:16.612135 7905 tablet_bootstrap.cc:654] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:16.612198 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.611811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:42347)
0620 18:30:16.611973 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.612063 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:30:16.613405 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.612967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:39793)
0620 18:30:16.613165 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.613255 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:30:16.615309 7800 heartbeater.cc:503] Master 127.0.97.61:35353 was elected leader, sending a full tablet report...
I20260620 18:30:16.621025 7906 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Bootstrap starting.
I20260620 18:30:16.621273 7905 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: No bootstrap required, opened a new log
I20260620 18:30:16.621433 7905 ts_tablet_manager.cc:1397] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260620 18:30:16.622404 7905 raft_consensus.cc:348] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.622758 7905 raft_consensus.cc:374] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:16.622901 7905 raft_consensus.cc:729] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47b6ef6f62f4aa0bc3721b1b3c3a3b1, State: Initialized, Role: FOLLOWER
I20260620 18:30:16.623037 7906 tablet_bootstrap.cc:654] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:16.623219 7905 consensus_queue.cc:260] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.624100 7905 ts_tablet_manager.cc:1428] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:30:16.627456 7512 heartbeater.cc:503] Master 127.0.97.61:35353 was elected leader, sending a full tablet report...
I20260620 18:30:16.628252 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.624789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:49303)
0620 18:30:16.627959 (+ 3170us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.628084 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3046}
I20260620 18:30:16.629287 7906 tablet_bootstrap.cc:492] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: No bootstrap required, opened a new log
I20260620 18:30:16.629437 7906 ts_tablet_manager.cc:1397] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 18:30:16.629794 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.629413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:39351)
0620 18:30:16.629561 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.629655 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:30:16.630213 7906 raft_consensus.cc:348] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.630391 7906 raft_consensus.cc:374] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:16.630498 7906 raft_consensus.cc:729] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0ae4d34fac454995aa3508ec23357e, State: Initialized, Role: FOLLOWER
I20260620 18:30:16.630764 7906 consensus_queue.cc:260] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.631460 7906 ts_tablet_manager.cc:1428] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:30:16.633193 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.632762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:57779)
0620 18:30:16.632974 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.633055 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 18:30:16.634086 7658 heartbeater.cc:503] Master 127.0.97.61:35353 was elected leader, sending a full tablet report...
I20260620 18:30:16.634197 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.633824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:45667)
0620 18:30:16.633973 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.634047 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:30:16.682473 7911 raft_consensus.cc:482] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:16.682729 7911 raft_consensus.cc:504] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.684576 7911 leader_election.cc:288] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153), a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:16.691632 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.684827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44153 (local address 127.0.97.2:34861)
0620 18:30:16.685056 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.685079 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:16.685239 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:16.685693 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:16.685701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:16.685738 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:16.686091 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.686105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.687333 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.687337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.689539 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.689559 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.690857 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.690865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.691067 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.691112 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:30:16.691188 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:30:16.691282 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":8}
I20260620 18:30:16.691658 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.685015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34861 (local address 127.0.97.3:44153)
0620 18:30:16.685220 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.685227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.685249 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:30:16.685425 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.685433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.685551 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.685653 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:16.686238 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.687137 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.689706 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.690656 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.690754 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.690828 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.691239 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.691378 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:30:16.691906 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.685248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52945 (local address 127.0.97.1:37701)
0620 18:30:16.685540 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.685548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.685980 (+ 432us) server_negotiation.cc:407] Connection header received
0620 18:30:16.686220 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.686224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.686306 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.686410 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:16.686912 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.687793 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.690019 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.690825 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.690953 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.691053 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.691388 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.691554 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 18:30:16.691941 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.685151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37701 (local address 127.0.97.2:52945)
0620 18:30:16.685874 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.685892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:16.686022 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:16.686435 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:16.686439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:16.686460 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:16.686742 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.686750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.687945 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.687951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.689871 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:16.689880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.691039 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.691043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:16.691221 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.691254 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:30:16.691342 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:30:16.691539 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":29,"thread_start_us":338,"threads_started":1}
I20260620 18:30:16.692497 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907113bdcee40d8bb5e7d4d95ce307e" candidate_uuid: "3f0ae4d34fac454995aa3508ec23357e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" is_pre_election: true
I20260620 18:30:16.692726 7465 raft_consensus.cc:2454] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f0ae4d34fac454995aa3508ec23357e in term 0.
I20260620 18:30:16.692644 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907113bdcee40d8bb5e7d4d95ce307e" candidate_uuid: "3f0ae4d34fac454995aa3508ec23357e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a08f8b79534940a66de768c7ca02a5" is_pre_election: true
I20260620 18:30:16.693080 7753 raft_consensus.cc:2454] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f0ae4d34fac454995aa3508ec23357e in term 0.
I20260620 18:30:16.693189 7547 leader_election.cc:302] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0ae4d34fac454995aa3508ec23357e, a47b6ef6f62f4aa0bc3721b1b3c3a3b1; no voters:
I20260620 18:30:16.693593 7911 raft_consensus.cc:2781] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:16.693739 7911 raft_consensus.cc:482] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:16.693853 7911 raft_consensus.cc:3037] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:16.696048 7911 raft_consensus.cc:504] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.696580 7911 leader_election.cc:288] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [CANDIDATE]: Term 1 election: Requested vote from peers 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153), a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701)
I20260620 18:30:16.697180 7753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907113bdcee40d8bb5e7d4d95ce307e" candidate_uuid: "3f0ae4d34fac454995aa3508ec23357e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a08f8b79534940a66de768c7ca02a5"
I20260620 18:30:16.697242 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907113bdcee40d8bb5e7d4d95ce307e" candidate_uuid: "3f0ae4d34fac454995aa3508ec23357e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1"
I20260620 18:30:16.697395 7753 raft_consensus.cc:3037] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:16.697433 7465 raft_consensus.cc:3037] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:16.700343 7465 raft_consensus.cc:2454] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f0ae4d34fac454995aa3508ec23357e in term 1.
I20260620 18:30:16.700343 7753 raft_consensus.cc:2454] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f0ae4d34fac454995aa3508ec23357e in term 1.
I20260620 18:30:16.700728 7547 leader_election.cc:302] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0ae4d34fac454995aa3508ec23357e, a47b6ef6f62f4aa0bc3721b1b3c3a3b1; no voters:
I20260620 18:30:16.701017 7911 raft_consensus.cc:2781] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:16.701469 7911 raft_consensus.cc:686] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 1 LEADER]: Becoming Leader. State: Replica: 3f0ae4d34fac454995aa3508ec23357e, State: Running, Role: LEADER
I20260620 18:30:16.702018 7911 consensus_queue.cc:237] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } }
I20260620 18:30:16.702443 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.702013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:38233)
0620 18:30:16.702196 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.702293 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:30:16.705265 7141 catalog_manager.cc:5495] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3f0ae4d34fac454995aa3508ec23357e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f0ae4d34fac454995aa3508ec23357e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 } health_report { overall_health: UNKNOWN } } }
W20260620 18:30:16.743961 7659 tablet.cc:2369] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:16.750535 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.745280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.97.2:33521)
0620 18:30:16.745475 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.745479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.745496 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:16.745677 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.745681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.745759 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.745885 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:16.746208 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.746915 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.748422 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.748853 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.748952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.749337 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:16.749861 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:16.749963 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.750135 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.750200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:30:16.754782 7465 raft_consensus.cc:1261] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Refusing update from remote peer 3f0ae4d34fac454995aa3508ec23357e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:16.755072 7753 raft_consensus.cc:1261] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Refusing update from remote peer 3f0ae4d34fac454995aa3508ec23357e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:16.755539 7913 consensus_queue.cc:1035] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47b6ef6f62f4aa0bc3721b1b3c3a3b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:16.755896 7911 consensus_queue.cc:1035] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a08f8b79534940a66de768c7ca02a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:16.758606 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.758037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:52067)
0620 18:30:16.758313 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.758418 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 18:30:16.759392 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.758944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:47309)
0620 18:30:16.759149 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.759232 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 18:30:16.762773 7916 mvcc.cc:205] Tried to move back new op lower bound from 7298990967820664832 to 7298990967615406080. Current Snapshot: MvccSnapshot[applied={T|T < 7298990967820664832}]
I20260620 18:30:16.764286 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.763759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.3:54459)
0620 18:30:16.763998 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.764092 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 18:30:16.766312 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.765823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:58551)
0620 18:30:16.765994 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.766118 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:30:16.774075 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.773603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.2:56803)
0620 18:30:16.773736 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.773829 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:30:16.777825 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.777245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.97.1:52299)
0620 18:30:16.777550 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:16.777635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 18:30:16.781244 7142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:16.781479 7142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:16.786772 7142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39f59276d8d74d2f891944e602f2c4ac: Sending DeleteTablet for 3 replicas of tablet d907113bdcee40d8bb5e7d4d95ce307e
I20260620 18:30:16.787739 7733 tablet_service.cc:1514] Processing DeleteTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:16 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:38690
I20260620 18:30:16.787901 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:16 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52420
I20260620 18:30:16.788543 7926 tablet_replica.cc:330] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: stopping tablet replica
I20260620 18:30:16.788777 7926 raft_consensus.cc:2229] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:16.788806 7927 tablet_replica.cc:330] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: stopping tablet replica
I20260620 18:30:16.789006 7927 raft_consensus.cc:2229] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:16.789038 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet d907113bdcee40d8bb5e7d4d95ce307e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:16 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33154
I20260620 18:30:16.789146 7926 raft_consensus.cc:2258] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:16.789710 7927 raft_consensus.cc:2258] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:16.790338 7926 ts_tablet_manager.cc:1905] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:16.790341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7319
I20260620 18:30:16.790872 7927 ts_tablet_manager.cc:1905] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:16.796798 7926 ts_tablet_manager.cc:1918] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:16.797019 7926 log.cc:1196] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/d907113bdcee40d8bb5e7d4d95ce307e
I20260620 18:30:16.797537 7926 ts_tablet_manager.cc:1939] T d907113bdcee40d8bb5e7d4d95ce307e P 41a08f8b79534940a66de768c7ca02a5: Deleting consensus metadata
I20260620 18:30:16.798671 7129 catalog_manager.cc:4841] TS 41a08f8b79534940a66de768c7ca02a5 (127.0.97.3:44153): tablet d907113bdcee40d8bb5e7d4d95ce307e (table test-table [id=cb270b054c754a8f81943ba0eb787f65]) successfully deleted
I20260620 18:30:16.798872 7927 ts_tablet_manager.cc:1918] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:16.799018 7927 log.cc:1196] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/d907113bdcee40d8bb5e7d4d95ce307e
I20260620 18:30:16.799631 7927 ts_tablet_manager.cc:1939] T d907113bdcee40d8bb5e7d4d95ce307e P 3f0ae4d34fac454995aa3508ec23357e: Deleting consensus metadata
I20260620 18:30:16.800954 7129 catalog_manager.cc:4841] TS 3f0ae4d34fac454995aa3508ec23357e (127.0.97.2:33521): tablet d907113bdcee40d8bb5e7d4d95ce307e (table test-table [id=cb270b054c754a8f81943ba0eb787f65]) successfully deleted
I20260620 18:30:16.804023 7928 tablet_replica.cc:330] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: stopping tablet replica
I20260620 18:30:16.804272 7928 raft_consensus.cc:2229] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:16.804628 7928 raft_consensus.cc:2258] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:16.805969 7928 ts_tablet_manager.cc:1905] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:16.812796 7928 ts_tablet_manager.cc:1918] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:16.812940 7928 log.cc:1196] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/d907113bdcee40d8bb5e7d4d95ce307e
I20260620 18:30:16.813354 7928 ts_tablet_manager.cc:1939] T d907113bdcee40d8bb5e7d4d95ce307e P a47b6ef6f62f4aa0bc3721b1b3c3a3b1: Deleting consensus metadata
I20260620 18:30:16.814447 7129 catalog_manager.cc:4841] TS a47b6ef6f62f4aa0bc3721b1b3c3a3b1 (127.0.97.1:37701): tablet d907113bdcee40d8bb5e7d4d95ce307e (table test-table [id=cb270b054c754a8f81943ba0eb787f65]) successfully deleted
I20260620 18:30:16.815673 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.793497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.97.1:37701)
0620 18:30:16.793675 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.793681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.800003 (+ 6322us) server_negotiation.cc:407] Connection header received
0620 18:30:16.800274 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.800281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.800362 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.800446 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:16.802441 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.803948 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.805623 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.806247 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.806396 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.809289 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:16.809306 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:16.809376 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:16.809428 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:16.812631 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:16.813398 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:16.813409 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:16.813413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:16.813571 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:16.813961 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:16.813968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:16.813971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:16.814457 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:16.814758 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.815067 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.815286 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:30:16.816285 7505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:16.924597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7319
I20260620 18:30:16.943697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7522
I20260620 18:30:16.960870 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:16.944540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.97.2:33521)
0620 18:30:16.944722 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:16.944728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:16.944748 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:30:16.944889 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:16.944895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:16.945002 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:16.945112 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:16.946724 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.948210 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.949806 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:16.950568 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:16.950716 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:16.953517 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:16.953535 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:16.953558 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:16.953602 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:16.957916 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:16.958594 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:16.958605 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:16.958611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:16.958807 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:16.959174 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:16.959180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:16.959184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:16.959676 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:16.960028 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:16.960182 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:30:16.960434 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:30:16.961656 7651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:17.036926 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:17.036517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55668)
0620 18:30:17.036691 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:17.036776 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:30:17.037954 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:17.037553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45127 (local address 127.0.0.1:55682)
0620 18:30:17.037702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:17.037783 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:30:17.081650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7522
I20260620 18:30:17.100549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7669
I20260620 18:30:17.115146 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:17.101333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.97.3:44153)
0620 18:30:17.101538 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:17.101544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:17.101566 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:30:17.101694 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:17.101702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:17.101796 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:17.101902 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:17.103180 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:17.103959 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:17.105180 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:17.105598 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:17.105690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:17.108701 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:17.108716 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:17.108733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:17.108784 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:17.112451 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:17.113115 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:17.113122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:17.113124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:17.113276 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:17.113652 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:17.113657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:17.113659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:17.114053 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:17.114284 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:17.114612 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:30:17.114769 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:30:17.116096 7793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:17.234246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7669
I20260620 18:30:17.255627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7005
I20260620 18:30:17.256716 7172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:17.380604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7005
W20260620 18:30:17.401285 7315 connection.cc:535] server connection from 127.0.0.1:33386 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:30:17.401691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7106
I20260620 18:30:17.402740 7362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:17.515823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7106
2026-06-20T18:30:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11251 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:17 dist-test-slave-mcnm krb5kdc[7937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:17 dist-test-slave-mcnm krb5kdc[7937](info): set up 2 sockets
Jun 20 18:30:17 dist-test-slave-mcnm krb5kdc[7937](info): commencing operation
krb5kdc: starting...
W20260620 18:30:18.609031 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:18 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 18:30:18.832387 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44251
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:19.149359 7950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:19.149631 7950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:19.149783 7950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:19.156898 7950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:19.156998 7950 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:30:19.157076 7950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:19.157191 7950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:19.157297 7950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:19.157400 7950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:19.166999 7950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44251
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:19.168282 7950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:19.169588 7950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:19.181967 7955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:19.182039 7956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:19.182057 7958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:19.182487 7950 server_base.cc:1034] running on GCE node
I20260620 18:30:19.183390 7950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:19.184789 7950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:19.186031 7950 hybrid_clock.cc:648] HybridClock initialized: now 1781980219185954 us; error 52 us; skew 500 ppm
I20260620 18:30:19.186372 7950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:19 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:19.189462 7950 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 18:30:19.190889 7950 webserver.cc:460] Webserver started at http://127.0.97.62:37431/ using document root <none> and password file <none>
I20260620 18:30:19.191677 7950 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:19.191816 7950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:19.192160 7950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:19.195060 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "dc6898f32fb9410ab9b0a7a4c77428ee"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "4cf28d43b5365d53bfe822c587ac7df1"
server_key_iv: "4d4399e56e8cf29564378291b59437ee"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.195909 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "dc6898f32fb9410ab9b0a7a4c77428ee"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "4cf28d43b5365d53bfe822c587ac7df1"
server_key_iv: "4d4399e56e8cf29564378291b59437ee"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.201042 7950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:30:19.204595 7965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:19.205968 7950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:30:19.206166 7950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "dc6898f32fb9410ab9b0a7a4c77428ee"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "4cf28d43b5365d53bfe822c587ac7df1"
server_key_iv: "4d4399e56e8cf29564378291b59437ee"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.206395 7950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:19.221293 7950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:19.225389 7950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:19.225680 7950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:19.246568 7950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44251
I20260620 18:30:19.246589 8026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44251 every 8 connection(s)
I20260620 18:30:19.248100 7950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:19.251952 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:19.253443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 7950
I20260620 18:30:19.253672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:19.254313 388 external_mini_cluster.cc:1428] Setting key 66d8a7699f1c777995c208efad8657db
I20260620 18:30:19.265116 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Bootstrap starting.
I20260620 18:30:19.268778 8027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:19.269752 8027 log.cc:824] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:30:19 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:19.272436 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: No bootstrap required, opened a new log
I20260620 18:30:19.278151 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:19.256302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.97.62:44251)
0620 18:30:19.257142 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:19.257158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:19.257208 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:30:19.259661 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:19.259697 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:19.260199 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:19.260647 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:19.261913 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.263981 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.265814 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.266848 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.267051 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:19.271089 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:19.271097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:19.271113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:19.271148 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:19.274254 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.274924 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.274931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.274939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.275112 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.275491 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.275495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.275499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.276025 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:19.276349 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:19.276783 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:30:19.277178 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":350,"threads_started":1}
I20260620 18:30:19.280319 8027 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } }
I20260620 18:30:19.280730 8027 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:19.280838 8027 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6898f32fb9410ab9b0a7a4c77428ee, State: Initialized, Role: FOLLOWER
I20260620 18:30:19.281450 8027 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } }
I20260620 18:30:19.281651 8027 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:19.281829 8027 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:19.281989 8027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:19.284106 8027 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } }
I20260620 18:30:19.284585 8027 leader_election.cc:302] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6898f32fb9410ab9b0a7a4c77428ee; no voters:
I20260620 18:30:19.285158 8027 leader_election.cc:288] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:19.285320 8032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:19.286402 8032 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [term 1 LEADER]: Becoming Leader. State: Replica: dc6898f32fb9410ab9b0a7a4c77428ee, State: Running, Role: LEADER
I20260620 18:30:19.286993 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } }
I20260620 18:30:19.287246 8027 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:19.289711 8034 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6898f32fb9410ab9b0a7a4c77428ee. Latest consensus state: current_term: 1 leader_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } } }
I20260620 18:30:19.290100 8034 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:19.290628 8033 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6898f32fb9410ab9b0a7a4c77428ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44251 } } }
I20260620 18:30:19.290896 8033 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:19.291810 8040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:19.297003 8040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:19.305110 8040 catalog_manager.cc:1347] Generated new cluster ID: d0f211d07fe4400da3180b9b0340a525
I20260620 18:30:19.305219 8040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:19.321784 8040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:19.323563 8040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:19.336891 8040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Generated new TSK 0
I20260620 18:30:19.337834 8040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 18:30:19.391623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44251
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:19.712529 8053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:19.712798 8053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:19.712893 8053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:19.720252 8053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:19.720359 8053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:19.720454 8053 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:30:19.720535 8053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:19.730677 8053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:19.731923 8053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:19.733183 8053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:19.745740 8059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:19.745929 8061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:19.745960 8058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:19.746480 8053 server_base.cc:1034] running on GCE node
I20260620 18:30:19.747069 8053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:19.748035 8053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:19.749243 8053 hybrid_clock.cc:648] HybridClock initialized: now 1781980219749175 us; error 37 us; skew 500 ppm
I20260620 18:30:19.749562 8053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:19 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:19.752683 8053 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 18:30:19.753947 8053 webserver.cc:460] Webserver started at http://127.0.97.1:32983/ using document root <none> and password file <none>
I20260620 18:30:19.754565 8053 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:19.754648 8053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:19.754899 8053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:19.757746 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "ef222a6038412add4bfe070480be9531"
server_key_iv: "1f11486e54f773ccd3f087f7fa71dd84"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.758915 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "ef222a6038412add4bfe070480be9531"
server_key_iv: "1f11486e54f773ccd3f087f7fa71dd84"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.764145 8053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 18:30:19.767331 8068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:19.768743 8053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:19.768939 8053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
format_stamp: "Formatted at 2026-06-20 18:30:19 on dist-test-slave-mcnm"
server_key: "ef222a6038412add4bfe070480be9531"
server_key_iv: "1f11486e54f773ccd3f087f7fa71dd84"
server_key_version: "encryptionkey@0"
I20260620 18:30:19.769196 8053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:19.792780 8053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:19.797106 8053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:19.797336 8053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:19.799863 8053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:19.799973 8053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:19.800100 8053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:19.800180 8053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:30:19 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:19.823668 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:19.800506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35929 (local address 127.0.97.62:44251)
0620 18:30:19.802384 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:19.802391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:19.802548 (+ 157us) server_negotiation.cc:407] Connection header received
0620 18:30:19.804597 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:19.804605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:19.804715 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:19.804851 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:19.807219 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.808530 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.811220 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.811706 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.811857 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:19.817312 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:19.817320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:19.817328 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:19.817374 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:19.820702 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.821396 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.821401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.821405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.821526 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.821977 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.821982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.821986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.822261 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:19.822400 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:19.823050 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 18:30:19.823274 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1762,"thread_start_us":1661,"threads_started":1}
I20260620 18:30:19.825019 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:19.801362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.1:35929)
0620 18:30:19.802407 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:19.802446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:19.804246 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:19.805106 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:19.805120 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:19.805987 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:19.806980 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:19.807018 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.808709 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.808717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:19.811074 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:19.811085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.812028 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.812038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:19.812399 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:19.813380 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:19.813420 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:19.817028 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:19.820835 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:19.820846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:19.820864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:19.821272 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:19.821638 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:19.821643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:19.821647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:19.821857 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:19.822388 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:19.822395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:19.822722 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:30:19.822978 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:30:19.823457 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":453,"threads_started":1}
I20260620 18:30:19.848419 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:19.834520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59637 (local address 127.0.97.62:44251)
0620 18:30:19.834707 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:19.834714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:19.835458 (+ 744us) server_negotiation.cc:407] Connection header received
0620 18:30:19.835701 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:19.835707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:19.835792 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:19.835912 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:19.837016 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.837703 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.839067 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.839894 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.840036 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:19.842554 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:19.842558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:19.842561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:19.842588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:19.845763 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.846442 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.846449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.846454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.846584 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:19.846943 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:19.846948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:19.846951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:19.847204 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:19.847348 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:19.847835 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:30:19.848060 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:30:19.848639 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:19.834588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.1:59637)
0620 18:30:19.835331 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:19.835357 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:19.835541 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:19.836055 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:19.836058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:19.836583 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:19.836871 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:19.836879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.837829 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.837836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:19.838885 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:19.838901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:19.840421 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:19.840431 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:19.840703 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:19.841405 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:19.841422 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:19.842345 (+ 923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:19.845883 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:19.845890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:19.845894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:19.846315 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:19.846698 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:19.846703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:19.846707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:19.846831 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:19.847326 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:19.847331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:19.847508 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 18:30:19.847739 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:30:19.848088 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":386,"threads_started":1}
I20260620 18:30:19.854003 8053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42711
I20260620 18:30:19.854022 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42711 every 8 connection(s)
I20260620 18:30:19.855161 8053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:19.856952 8189 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44251
I20260620 18:30:19.857201 8189 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:19.857782 8189 heartbeater.cc:511] Master 127.0.97.62:44251 requested a full tablet report, sending...
I20260620 18:30:19.859665 7982 ts_manager.cc:194] Registered new tserver with Master: eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711)
I20260620 18:30:19.861025 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35929
I20260620 18:30:19.861845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8053
I20260620 18:30:19.862033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:19.862660 388 external_mini_cluster.cc:1428] Setting key c508004a126b00f761d42d2eaa94bf1b
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 18:30:19.907970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44251
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:20.225063 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:20.225368 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:20.225529 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:20.232650 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:20.232784 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:20.232929 8193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:30:20.233063 8193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:20.242388 8193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:20.243619 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:20.244936 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:20.259024 8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:20.259039 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:20.259414 8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:20.259821 8193 server_base.cc:1034] running on GCE node
I20260620 18:30:20.260473 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:20.261317 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:20.262550 8193 hybrid_clock.cc:648] HybridClock initialized: now 1781980220262478 us; error 54 us; skew 500 ppm
I20260620 18:30:20.262832 8193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:20.265877 8193 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 18:30:20.267227 8193 webserver.cc:460] Webserver started at http://127.0.97.2:34671/ using document root <none> and password file <none>
I20260620 18:30:20.268105 8193 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:20.268218 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:20.268617 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:20.272128 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "e11a3b34c18243a99b66822123836bcb"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "2df8bed4e60f90761e78839187ca56e2"
server_key_iv: "9c5f7d597c33fb34238e99d8a7243440"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.272976 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e11a3b34c18243a99b66822123836bcb"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "2df8bed4e60f90761e78839187ca56e2"
server_key_iv: "9c5f7d597c33fb34238e99d8a7243440"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.277861 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:20.281095 8208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:20.282276 8193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:30:20.282436 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "e11a3b34c18243a99b66822123836bcb"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "2df8bed4e60f90761e78839187ca56e2"
server_key_iv: "9c5f7d597c33fb34238e99d8a7243440"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.282639 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:20.306947 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:20.311333 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:20.311627 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:20.314301 8193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:20.314416 8193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:20.314558 8193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:20.314630 8193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:20.336010 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.314725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59575 (local address 127.0.97.62:44251)
0620 18:30:20.314933 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.314941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.316284 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 18:30:20.318428 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.318433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.318527 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.318647 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.320802 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.321844 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.324498 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.324919 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.325056 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.329496 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:20.329501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:20.329504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:20.329534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:20.332475 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.333570 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.333578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.333584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.333749 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.334325 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.334331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.334336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.334632 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:20.334791 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.335436 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.335673 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:30:20.336956 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.315335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.2:59575)
0620 18:30:20.316119 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.316164 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.318158 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.318900 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.318909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.319761 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:20.320620 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.320650 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.322006 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.322012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.324339 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.324351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.325148 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.325156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.325361 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.326122 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:20.326147 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:20.329244 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:20.332648 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.332664 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.332682 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.333416 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.333910 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.333913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.333916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.334166 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.334762 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:20.334770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:20.335077 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 18:30:20.335328 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 18:30:20.335868 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":347,"threads_started":1}
I20260620 18:30:20.359134 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.342956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.2:36329)
0620 18:30:20.343653 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.343668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.343769 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.344130 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.344133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.344502 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:20.344761 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.344769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.346452 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.346455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.347850 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.347866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.349225 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.349235 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.349459 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.350347 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:20.350381 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:20.351885 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:20.356061 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.356064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.356066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.356538 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.356961 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.356969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.356974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.357142 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.357858 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:20.357862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:20.358146 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 18:30:20.358316 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:30:20.358602 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":395,"threads_started":1}
I20260620 18:30:20.359869 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.342857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36329 (local address 127.0.97.62:44251)
0620 18:30:20.343004 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.343008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.343765 (+ 757us) server_negotiation.cc:407] Connection header received
0620 18:30:20.343859 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.343867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.343978 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.344101 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.344891 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.346311 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.348005 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.348778 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.348923 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.352095 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:20.352101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:20.352105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:20.352144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:20.355929 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.356679 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.356684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.356687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.356825 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.357258 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.357265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.357272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.357744 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:20.357880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.359449 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.359673 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:30:20.374148 8193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41799
I20260620 18:30:20.374161 8327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41799 every 8 connection(s)
I20260620 18:30:20.375365 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:20.377612 8328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44251
I20260620 18:30:20.377848 8328 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:20.378139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8193
I20260620 18:30:20.378301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:20.378641 8328 heartbeater.cc:511] Master 127.0.97.62:44251 requested a full tablet report, sending...
I20260620 18:30:20.378880 388 external_mini_cluster.cc:1428] Setting key 07d294fecc25ba5c3452a9bbade07cc8
I20260620 18:30:20.380151 7982 ts_manager.cc:194] Registered new tserver with Master: e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799)
I20260620 18:30:20.381161 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59575
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 18:30:20.427273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44251
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:20.740425 8332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:20.740689 8332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:20.740849 8332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:20.747972 8332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:20.748083 8332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:20.748220 8332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:30:20.748346 8332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:20.757839 8332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:20.759074 8332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:20.760393 8332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:20.775489 8340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:20.775550 8338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:20.775578 8337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:20.776232 8332 server_base.cc:1034] running on GCE node
I20260620 18:30:20.776747 8332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:20.777578 8332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:20.778785 8332 hybrid_clock.cc:648] HybridClock initialized: now 1781980220778736 us; error 26 us; skew 500 ppm
I20260620 18:30:20.779031 8332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:20.782130 8332 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 18:30:20.783497 8332 webserver.cc:460] Webserver started at http://127.0.97.3:37991/ using document root <none> and password file <none>
I20260620 18:30:20.784251 8332 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:20.784334 8332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:20.784680 8332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:20.787863 8332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "63b2ba0eeb68469e8aae336407b5713c"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "3564846a8e27879e093c9f9bb990c348"
server_key_iv: "c5e0ec8e0a7d16d5b78e39e5897be75d"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.788964 8332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "63b2ba0eeb68469e8aae336407b5713c"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "3564846a8e27879e093c9f9bb990c348"
server_key_iv: "c5e0ec8e0a7d16d5b78e39e5897be75d"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.794124 8332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:30:20.797462 8347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:20.798847 8332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:20.799052 8332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "63b2ba0eeb68469e8aae336407b5713c"
format_stamp: "Formatted at 2026-06-20 18:30:20 on dist-test-slave-mcnm"
server_key: "3564846a8e27879e093c9f9bb990c348"
server_key_iv: "c5e0ec8e0a7d16d5b78e39e5897be75d"
server_key_version: "encryptionkey@0"
I20260620 18:30:20.799295 8332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:20.819142 8332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:20.822973 8332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:20.823246 8332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:20.825786 8332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:20.825891 8332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:20.826042 8332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:20.826134 8332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:20.851466 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.826520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58159 (local address 127.0.97.62:44251)
0620 18:30:20.826712 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.826719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.828134 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 18:30:20.831237 (+ 3103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.831246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.831344 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.831502 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.833874 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.834937 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.837369 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.837917 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.838078 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.843548 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:20.843558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:20.843565 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:20.843621 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:20.847186 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.848061 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.848075 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.848079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.848221 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.848714 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.848719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.848723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.849157 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:20.849324 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.851067 (+ 1743us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.851225 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:30:20.852735 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.827061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.3:58159)
0620 18:30:20.827960 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.828007 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.830807 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.831838 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.831850 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.832697 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:20.833656 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.833685 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.835108 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.835113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.837230 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.837241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.838222 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.838233 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.838632 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.839614 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:20.839657 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:20.843273 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:20.847373 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.847384 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.847407 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.847923 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.848341 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.848346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.848350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.848566 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.849314 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:20.849322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:20.849873 (+ 551us) client_negotiation.cc:769] Sending connection context
0620 18:30:20.850196 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 18:30:20.850667 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260620 18:30:20.863585 8189 heartbeater.cc:503] Master 127.0.97.62:44251 was elected leader, sending a full tablet report...
I20260620 18:30:20.875976 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.861778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.97.3:51701)
0620 18:30:20.862531 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.862557 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.862775 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.863175 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.863179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.863783 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:20.864210 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.864223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.865302 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.865306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.866431 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.866441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.867120 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.867127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.867290 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.868018 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:20.868047 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:20.869154 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:20.873047 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.873051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.873057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.873697 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.874120 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:20.874126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:20.874131 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:20.874255 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:20.874806 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:20.874811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:20.875038 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 18:30:20.875240 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 18:30:20.875655 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":376,"threads_started":1}
I20260620 18:30:20.876137 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.861793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51701 (local address 127.0.97.62:44251)
0620 18:30:20.862008 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.862017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.862640 (+ 623us) server_negotiation.cc:407] Connection header received
0620 18:30:20.862922 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.862929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.863031 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.863149 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.864377 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.865116 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.866555 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.866943 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.867088 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.869387 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:20.869395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:20.869401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:20.869445 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:20.872886 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.873877 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.873882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.873886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.874008 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.874416 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.874421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.874424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.874684 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:20.874826 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.875381 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.875658 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:30:20.879822 8332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42467
I20260620 18:30:20.879851 8466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42467 every 8 connection(s)
I20260620 18:30:20.880990 8332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:20.882926 8467 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44251
I20260620 18:30:20.883137 8467 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:20.883805 8467 heartbeater.cc:511] Master 127.0.97.62:44251 requested a full tablet report, sending...
I20260620 18:30:20.885006 7982 ts_manager.cc:194] Registered new tserver with Master: 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:20.885795 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58159
I20260620 18:30:20.886968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8332
I20260620 18:30:20.887181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:20.887846 388 external_mini_cluster.cc:1428] Setting key 1f4eae40a40dadb42316b5b193bae962
I20260620 18:30:20.890606 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:30:20.939113 8077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:20.928955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.0.97.62:44251)
0620 18:30:20.929188 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.929196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.929372 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:30:20.929605 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.929612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.929685 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.929792 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.931114 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.931907 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.933129 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.933541 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.933681 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.938883 (+ 5202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59280: BlockingRecv error: recv got EOF from 127.0.0.1:59280 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:30:20 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:20.956614 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.942519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.0.97.62:44251)
0620 18:30:20.942761 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.942768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.942963 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:30:20.943068 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.943071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.943148 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.943237 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:20.944479 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.945173 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.946356 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.946767 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.946862 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.950790 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:20.950796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:20.950799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:20.950845 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:20.953774 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.954470 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.954477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.954481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.954612 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:20.955030 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:20.955037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:20.955041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:20.955330 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:20.955674 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.956027 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.956261 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:30:20.960760 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:20.967798 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:20.992928 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.979881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.0.1:52290)
0620 18:30:20.981230 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.981262 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.981420 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.982203 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.982210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.982241 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:20.982716 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.982740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.985871 (+ 3131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.985880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.989003 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.989024 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.991499 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.991526 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.991836 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.991914 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 18:30:20.992040 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:30:20.992343 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"spinlock_wait_cycles":896,"thread_start_us":961,"threads_started":1}
I20260620 18:30:20.994578 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.980152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.97.2:41799)
0620 18:30:20.980853 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.980864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.981373 (+ 509us) server_negotiation.cc:407] Connection header received
0620 18:30:20.981667 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.981677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.981967 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.983554 (+ 1587us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:20.983801 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.985685 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.989304 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.991221 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.993568 (+ 2347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:20.993849 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:20.993932 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 18:30:20.994171 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":347,"threads_started":1}
I20260620 18:30:20.999042 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef87f05533649a7a736061f91e179fe (DEFAULT_TABLE table=test-table [id=8a98c7caa5134743a47fbe69ff43cb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.000520 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef87f05533649a7a736061f91e179fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.002687 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.982192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42711 (local address 127.0.0.1:36926)
0620 18:30:20.993224 (+ 11032us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.993239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.993360 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.996182 (+ 2822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.996186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.996211 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:20.996377 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.996392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.998176 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.998181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:20.999476 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.999494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.001781 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.001787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.002029 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.002071 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.002198 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.002367 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10889,"thread_start_us":12638,"threads_started":1}
I20260620 18:30:21.002770 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.982651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.0.97.1:42711)
0620 18:30:20.995604 (+ 12953us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.995613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.995640 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:30:20.995744 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.995751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.996003 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.996203 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:20.996558 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.998003 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.999934 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.001564 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.001730 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.001933 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.002259 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.002454 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12777,"thread_start_us":12661,"threads_started":1}
I20260620 18:30:21.005591 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.981590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42467 (local address 127.0.0.1:43220)
0620 18:30:20.982148 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:20.982166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:20.982279 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:20.997408 (+ 15129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:20.997414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:20.997443 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:20.997846 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:20.997870 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.999918 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:20.999924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.002238 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.002254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.004736 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.004743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.004966 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.005013 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.005122 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.005260 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
I20260620 18:30:21.005736 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:20.982578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.0.97.3:42467)
0620 18:30:20.996854 (+ 14276us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:20.996862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:20.996895 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:30:20.996981 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:20.996987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:20.997201 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:20.997375 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:20.998111 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:20.999768 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.002438 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.004229 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.004374 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.004596 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.005190 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.005368 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14107,"thread_start_us":13926,"threads_started":1}
I20260620 18:30:21.006764 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef87f05533649a7a736061f91e179fe (DEFAULT_TABLE table=test-table [id=8a98c7caa5134743a47fbe69ff43cb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.008323 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef87f05533649a7a736061f91e179fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.009665 8397 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef87f05533649a7a736061f91e179fe (DEFAULT_TABLE table=test-table [id=8a98c7caa5134743a47fbe69ff43cb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.011149 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef87f05533649a7a736061f91e179fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.014014 8491 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Bootstrap starting.
I20260620 18:30:21.017081 8491 tablet_bootstrap.cc:654] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.018122 8491 log.cc:824] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:21.023216 8491 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: No bootstrap required, opened a new log
I20260620 18:30:21.023530 8491 ts_tablet_manager.cc:1397] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:30:21.024055 8493 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Bootstrap starting.
I20260620 18:30:21.027879 8493 tablet_bootstrap.cc:654] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.027923 8494 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Bootstrap starting.
I20260620 18:30:21.029248 8493 log.cc:824] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:21.031253 8494 tablet_bootstrap.cc:654] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.032407 8494 log.cc:824] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:21.033057 8493 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: No bootstrap required, opened a new log
I20260620 18:30:21.033334 8493 ts_tablet_manager.cc:1397] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:30:21.034649 8491 raft_consensus.cc:348] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.035013 8494 tablet_bootstrap.cc:492] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: No bootstrap required, opened a new log
I20260620 18:30:21.035127 8491 raft_consensus.cc:374] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.035231 8491 raft_consensus.cc:729] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.035283 8494 ts_tablet_manager.cc:1397] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 18:30:21.035889 8491 consensus_queue.cc:260] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.037000 8328 heartbeater.cc:503] Master 127.0.97.62:44251 was elected leader, sending a full tablet report...
I20260620 18:30:21.037667 8491 ts_tablet_manager.cc:1428] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:30:21.044516 8493 raft_consensus.cc:348] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.044981 8493 raft_consensus.cc:374] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.045096 8493 raft_consensus.cc:729] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.045735 8493 consensus_queue.cc:260] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.047603 8493 ts_tablet_manager.cc:1428] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260620 18:30:21.047430 8494 raft_consensus.cc:348] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.047919 8494 raft_consensus.cc:374] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.048054 8494 raft_consensus.cc:729] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.048684 8494 consensus_queue.cc:260] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.049860 8467 heartbeater.cc:503] Master 127.0.97.62:44251 was elected leader, sending a full tablet report...
I20260620 18:30:21.050490 8494 ts_tablet_manager.cc:1428] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260620 18:30:21.107504 8190 tablet.cc:2369] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:30:21.127405 8329 tablet.cc:2369] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:30:21.132555 8468 tablet.cc:2369] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:21.173089 8499 raft_consensus.cc:482] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:21.173413 8499 raft_consensus.cc:504] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.174976 8499 leader_election.cc:288] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799), eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711)
I20260620 18:30:21.181429 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.175279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.97.3:54637)
0620 18:30:21.175500 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.175545 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.175692 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.176114 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.176121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.176153 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:21.176378 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.176389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.177700 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.177704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.179589 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.179603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.180905 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.180909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.181054 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.181078 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.181150 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.181236 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":25}
I20260620 18:30:21.181564 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.175498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54637 (local address 127.0.97.2:41799)
0620 18:30:21.175720 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.175726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.175747 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:30:21.175817 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.175822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.175917 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.176027 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:21.176552 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.177560 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.179743 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.180765 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.180897 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.180996 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.181196 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.181322 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:30:21.182359 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.175821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45375 (local address 127.0.97.1:42711)
0620 18:30:21.176016 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.176021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.176543 (+ 522us) server_negotiation.cc:407] Connection header received
0620 18:30:21.176689 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.176696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.176767 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.176860 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:21.177352 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.178135 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.180252 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.181415 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.181537 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.181667 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.182100 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.182200 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:30:21.182566 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.175790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42711 (local address 127.0.97.3:45375)
0620 18:30:21.176435 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.176454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.176562 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.176874 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.176876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.176893 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:21.177199 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.177206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.178262 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.178269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.180106 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.180116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.181804 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.181810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.181951 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.181974 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.182046 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.182186 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":25,"thread_start_us":334,"threads_started":1}
I20260620 18:30:21.182595 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef87f05533649a7a736061f91e179fe" candidate_uuid: "63b2ba0eeb68469e8aae336407b5713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11a3b34c18243a99b66822123836bcb" is_pre_election: true
I20260620 18:30:21.183108 8283 raft_consensus.cc:2454] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b2ba0eeb68469e8aae336407b5713c in term 0.
I20260620 18:30:21.183583 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef87f05533649a7a736061f91e179fe" candidate_uuid: "63b2ba0eeb68469e8aae336407b5713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" is_pre_election: true
I20260620 18:30:21.183795 8348 leader_election.cc:302] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, e11a3b34c18243a99b66822123836bcb; no voters:
I20260620 18:30:21.184096 8139 raft_consensus.cc:2454] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b2ba0eeb68469e8aae336407b5713c in term 0.
I20260620 18:30:21.184180 8499 raft_consensus.cc:2781] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:21.184358 8499 raft_consensus.cc:482] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:21.184464 8499 raft_consensus.cc:3037] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.186791 8499 raft_consensus.cc:504] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.187422 8499 leader_election.cc:288] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [CANDIDATE]: Term 1 election: Requested vote from peers e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799), eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711)
I20260620 18:30:21.187983 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef87f05533649a7a736061f91e179fe" candidate_uuid: "63b2ba0eeb68469e8aae336407b5713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
I20260620 18:30:21.187971 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef87f05533649a7a736061f91e179fe" candidate_uuid: "63b2ba0eeb68469e8aae336407b5713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11a3b34c18243a99b66822123836bcb"
I20260620 18:30:21.188180 8283 raft_consensus.cc:3037] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.188158 8139 raft_consensus.cc:3037] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.190285 8139 raft_consensus.cc:2454] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b2ba0eeb68469e8aae336407b5713c in term 1.
I20260620 18:30:21.190299 8283 raft_consensus.cc:2454] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b2ba0eeb68469e8aae336407b5713c in term 1.
I20260620 18:30:21.190644 8348 leader_election.cc:302] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, e11a3b34c18243a99b66822123836bcb; no voters:
I20260620 18:30:21.190937 8499 raft_consensus.cc:2781] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:21.191361 8499 raft_consensus.cc:686] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 1 LEADER]: Becoming Leader. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Running, Role: LEADER
I20260620 18:30:21.191879 8499 consensus_queue.cc:237] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } }
I20260620 18:30:21.195894 7982 catalog_manager.cc:5495] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "63b2ba0eeb68469e8aae336407b5713c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260620 18:30:21.228871 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.223251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.97.3:42467)
0620 18:30:21.223437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.223442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.223460 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:21.223564 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.223568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.223704 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.223904 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:21.224374 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.225190 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.226498 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.227003 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.227140 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.227625 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.228286 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.228445 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.228522 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.228598 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:30:21.236899 8283 raft_consensus.cc:1261] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Refusing update from remote peer 63b2ba0eeb68469e8aae336407b5713c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.236887 8139 raft_consensus.cc:1261] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Refusing update from remote peer 63b2ba0eeb68469e8aae336407b5713c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.237805 8501 consensus_queue.cc:1035] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.238219 8499 consensus_queue.cc:1035] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.250082 8506 mvcc.cc:205] Tried to move back new op lower bound from 7298990986176716800 to 7298990986006016000. Current Snapshot: MvccSnapshot[applied={T|T < 7298990986176716800}]
I20260620 18:30:21.251649 8504 mvcc.cc:205] Tried to move back new op lower bound from 7298990986176716800 to 7298990986006016000. Current Snapshot: MvccSnapshot[applied={T|T < 7298990986176716800}]
I20260620 18:30:21.272533 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:21.272804 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:21.277019 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Sending DeleteTablet for 3 replicas of tablet 6ef87f05533649a7a736061f91e179fe
I20260620 18:30:21.278156 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef87f05533649a7a736061f91e179fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 18:30:21.278327 8397 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef87f05533649a7a736061f91e179fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43220
I20260620 18:30:21.278863 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef87f05533649a7a736061f91e179fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36926
I20260620 18:30:21.279160 8515 tablet_replica.cc:330] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: stopping tablet replica
I20260620 18:30:21.279767 8515 raft_consensus.cc:2229] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:21.280329 8515 raft_consensus.cc:2258] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.281194 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:21.282078 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:21.282670 8515 ts_tablet_manager.cc:1905] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.289785 8515 ts_tablet_manager.cc:1918] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:21.290056 8515 log.cc:1196] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/6ef87f05533649a7a736061f91e179fe
I20260620 18:30:21.290653 8515 ts_tablet_manager.cc:1939] T 6ef87f05533649a7a736061f91e179fe P e11a3b34c18243a99b66822123836bcb: Deleting consensus metadata
I20260620 18:30:21.292768 7966 catalog_manager.cc:4841] TS e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799): tablet 6ef87f05533649a7a736061f91e179fe (table test-table [id=8a98c7caa5134743a47fbe69ff43cb99]) successfully deleted
I20260620 18:30:21.292968 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7d81ac51a41178923192b850cab90 (DEFAULT_TABLE table=test-table [id=beacc5d7087c4f78bc28e952ced3ad05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.293395 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7d81ac51a41178923192b850cab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.293767 8118 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7d81ac51a41178923192b850cab90 (DEFAULT_TABLE table=test-table [id=beacc5d7087c4f78bc28e952ced3ad05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.294252 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7d81ac51a41178923192b850cab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.297369 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7d81ac51a41178923192b850cab90 (DEFAULT_TABLE table=test-table [id=beacc5d7087c4f78bc28e952ced3ad05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.297878 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7d81ac51a41178923192b850cab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.302030 8493 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Bootstrap starting.
I20260620 18:30:21.303706 8493 tablet_bootstrap.cc:654] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.308126 8517 tablet_replica.cc:330] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: stopping tablet replica
I20260620 18:30:21.308717 8517 raft_consensus.cc:2229] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:21.308902 8491 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Bootstrap starting.
I20260620 18:30:21.310772 8491 tablet_bootstrap.cc:654] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.310746 8517 raft_consensus.cc:2258] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.312726 8494 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Bootstrap starting.
I20260620 18:30:21.312965 8517 ts_tablet_manager.cc:1905] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.313170 8491 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: No bootstrap required, opened a new log
I20260620 18:30:21.313290 8491 ts_tablet_manager.cc:1397] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:30:21.314039 8491 raft_consensus.cc:348] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.314244 8491 raft_consensus.cc:374] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.314317 8494 tablet_bootstrap.cc:654] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.314325 8491 raft_consensus.cc:729] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.314637 8491 consensus_queue.cc:260] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.315186 8491 ts_tablet_manager.cc:1428] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:30:21.315740 8516 tablet_replica.cc:330] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: stopping tablet replica
I20260620 18:30:21.316123 8516 raft_consensus.cc:2229] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:21.316706 8516 raft_consensus.cc:2258] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.321105 8516 ts_tablet_manager.cc:1905] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.321815 8517 ts_tablet_manager.cc:1918] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:21.321969 8517 log.cc:1196] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/6ef87f05533649a7a736061f91e179fe
I20260620 18:30:21.322485 8517 ts_tablet_manager.cc:1939] T 6ef87f05533649a7a736061f91e179fe P eb13bf58548b4a859ce01f94b36a9fd2: Deleting consensus metadata
I20260620 18:30:21.323866 8494 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: No bootstrap required, opened a new log
I20260620 18:30:21.324083 8494 ts_tablet_manager.cc:1397] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260620 18:30:21.324216 7966 catalog_manager.cc:4841] TS eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711): tablet 6ef87f05533649a7a736061f91e179fe (table test-table [id=8a98c7caa5134743a47fbe69ff43cb99]) successfully deleted
I20260620 18:30:21.325063 8494 raft_consensus.cc:348] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.325353 8494 raft_consensus.cc:374] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.325497 8494 raft_consensus.cc:729] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.325858 8494 consensus_queue.cc:260] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.328537 8493 tablet_bootstrap.cc:492] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: No bootstrap required, opened a new log
I20260620 18:30:21.328652 8516 ts_tablet_manager.cc:1918] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:21.328680 8493 ts_tablet_manager.cc:1397] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.011s
I20260620 18:30:21.328819 8516 log.cc:1196] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/6ef87f05533649a7a736061f91e179fe
I20260620 18:30:21.329185 8494 ts_tablet_manager.cc:1428] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:30:21.329324 8516 ts_tablet_manager.cc:1939] T 6ef87f05533649a7a736061f91e179fe P 63b2ba0eeb68469e8aae336407b5713c: Deleting consensus metadata
I20260620 18:30:21.329455 8493 raft_consensus.cc:348] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.329697 8493 raft_consensus.cc:374] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.329802 8493 raft_consensus.cc:729] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.330081 8493 consensus_queue.cc:260] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.330646 8493 ts_tablet_manager.cc:1428] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:30:21.330849 7966 catalog_manager.cc:4841] TS 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467): tablet 6ef87f05533649a7a736061f91e179fe (table test-table [id=8a98c7caa5134743a47fbe69ff43cb99]) successfully deleted
I20260620 18:30:21.345918 8497 raft_consensus.cc:482] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:21.346154 8497 raft_consensus.cc:504] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.348156 8497 leader_election.cc:288] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:21.355319 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.348495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57261 (local address 127.0.97.1:42711)
0620 18:30:21.348688 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.348692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.349129 (+ 437us) server_negotiation.cc:407] Connection header received
0620 18:30:21.349327 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.349332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.349421 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.349508 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:21.350427 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.351562 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.353581 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.354350 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.354497 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.354612 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.354906 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.355054 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:30:21.355322 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.348429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42711 (local address 127.0.97.2:57261)
0620 18:30:21.349017 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.349039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.349184 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.349697 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.349703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.349732 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:21.350191 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.350206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.351702 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.351708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.353423 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.353434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.354555 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.354561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.354728 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.354761 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.354857 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.354985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":372,"threads_started":1}
I20260620 18:30:21.355908 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7d81ac51a41178923192b850cab90" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" is_pre_election: true
I20260620 18:30:21.356168 8139 raft_consensus.cc:2454] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.356686 8209 leader_election.cc:302] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e11a3b34c18243a99b66822123836bcb, eb13bf58548b4a859ce01f94b36a9fd2; no voters:
I20260620 18:30:21.357101 8497 raft_consensus.cc:2781] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:21.357177 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.349529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35979 (local address 127.0.97.3:42467)
0620 18:30:21.349698 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.349704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.350336 (+ 632us) server_negotiation.cc:407] Connection header received
0620 18:30:21.350480 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.350484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.350543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.350644 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:21.351131 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.352177 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.354723 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.356185 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.356328 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.356428 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.356741 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.356889 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:30:21.357231 8497 raft_consensus.cc:482] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:21.357321 8497 raft_consensus.cc:3037] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.357604 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.349216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42467 (local address 127.0.97.2:35979)
0620 18:30:21.350225 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.350245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.350357 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.350657 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.350661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.350683 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:21.350962 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.350970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.352321 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.352327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.354497 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.354512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.356424 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.356430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.356583 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.356605 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.356716 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.356834 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":4,"thread_start_us":638,"threads_started":1}
I20260620 18:30:21.360386 8417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7d81ac51a41178923192b850cab90" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c" is_pre_election: true
I20260620 18:30:21.360875 8417 raft_consensus.cc:2454] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.361054 8497 raft_consensus.cc:504] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.361737 8497 leader_election.cc:288] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Requested vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:21.362129 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7d81ac51a41178923192b850cab90" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
I20260620 18:30:21.362171 8417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7d81ac51a41178923192b850cab90" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c"
I20260620 18:30:21.362324 8139 raft_consensus.cc:3037] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.362346 8417 raft_consensus.cc:3037] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.364591 8417 raft_consensus.cc:2454] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:21.364956 8209 leader_election.cc:302] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, e11a3b34c18243a99b66822123836bcb; no voters:
I20260620 18:30:21.365231 8497 raft_consensus.cc:2781] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:21.365211 8139 raft_consensus.cc:2454] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:21.365718 8497 raft_consensus.cc:686] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 1 LEADER]: Becoming Leader. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Running, Role: LEADER
I20260620 18:30:21.366086 8497 consensus_queue.cc:237] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.369443 7981 catalog_manager.cc:5495] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to e11a3b34c18243a99b66822123836bcb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:30:21 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:21.417282 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.402269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44251 (local address 127.0.0.1:59310)
0620 18:30:21.402989 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.403015 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.403182 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.403610 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.403615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.404248 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:21.404503 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.404512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.405983 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.405987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.406939 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.406949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.407932 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.407938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.408073 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.408821 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:21.408843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:21.411251 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:21.414957 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:21.414966 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:21.414972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:21.415353 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:21.415753 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:21.415756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:21.415758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:21.415882 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:21.416448 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:21.416454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:21.416606 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.416753 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.416935 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":394,"threads_started":1}
I20260620 18:30:21.417522 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.402266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.97.62:44251)
0620 18:30:21.402495 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.402502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.403100 (+ 598us) server_negotiation.cc:407] Connection header received
0620 18:30:21.403335 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.403341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.403418 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.403570 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:21.404639 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.405825 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.407070 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.407707 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.407830 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.411435 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:21.411442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:21.411447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:21.411482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:21.414807 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:21.415460 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:21.415466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:21.415470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:21.415643 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:21.415991 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:21.415996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:21.416000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:21.416325 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:21.416458 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.416869 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.417096 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:30:21.422456 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59310:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:30:21.423477 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:21.429877 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb8e3aae8c41bba27be3bbe314ff30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7308177e37d74db89c08827a79a659c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:30:21.429939 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb8e3aae8c41bba27be3bbe314ff30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7308177e37d74db89c08827a79a659c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:30:21.430423 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb8e3aae8c41bba27be3bbe314ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.430485 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb8e3aae8c41bba27be3bbe314ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.431313 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb8e3aae8c41bba27be3bbe314ff30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7308177e37d74db89c08827a79a659c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:30:21.431815 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb8e3aae8c41bba27be3bbe314ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.437390 8493 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: Bootstrap starting.
I20260620 18:30:21.437403 8491 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: Bootstrap starting.
I20260620 18:30:21.439014 8491 tablet_bootstrap.cc:654] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.439043 8493 tablet_bootstrap.cc:654] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.440132 8494 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: Bootstrap starting.
I20260620 18:30:21.441824 8494 tablet_bootstrap.cc:654] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.444077 8491 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: No bootstrap required, opened a new log
I20260620 18:30:21.444339 8491 ts_tablet_manager.cc:1397] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260620 18:30:21.445340 8491 raft_consensus.cc:348] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.445729 8491 raft_consensus.cc:374] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.445869 8491 raft_consensus.cc:729] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.446249 8491 consensus_queue.cc:260] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.446841 8497 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.447196 8497 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.447883 8491 ts_tablet_manager.cc:1428] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:30:21.450088 8493 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: No bootstrap required, opened a new log
I20260620 18:30:21.450279 8493 ts_tablet_manager.cc:1397] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260620 18:30:21.451195 8493 raft_consensus.cc:348] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.451506 8493 raft_consensus.cc:374] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.451665 8493 raft_consensus.cc:729] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.451959 8493 consensus_queue.cc:260] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.452839 8493 ts_tablet_manager.cc:1428] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:30:21.452598 8498 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.453002 8498 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.455955 8494 tablet_bootstrap.cc:492] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: No bootstrap required, opened a new log
I20260620 18:30:21.456241 8494 ts_tablet_manager.cc:1397] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260620 18:30:21.456982 8494 raft_consensus.cc:348] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.457324 8494 raft_consensus.cc:374] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.457417 8494 raft_consensus.cc:729] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.457696 8494 consensus_queue.cc:260] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.458061 8501 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.458235 8494 ts_tablet_manager.cc:1428] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:30:21.458367 8501 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.538683 8497 raft_consensus.cc:482] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:21.538934 8497 raft_consensus.cc:504] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.539738 8497 leader_election.cc:288] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:21.540269 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb8e3aae8c41bba27be3bbe314ff30" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" is_pre_election: true
I20260620 18:30:21.540539 8139 raft_consensus.cc:2454] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.540468 8417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb8e3aae8c41bba27be3bbe314ff30" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c" is_pre_election: true
I20260620 18:30:21.540776 8417 raft_consensus.cc:2454] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.540989 8209 leader_election.cc:302] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e11a3b34c18243a99b66822123836bcb, eb13bf58548b4a859ce01f94b36a9fd2; no voters:
I20260620 18:30:21.541352 8497 raft_consensus.cc:2781] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:21.541461 8497 raft_consensus.cc:482] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:21.541584 8497 raft_consensus.cc:3037] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.543769 8497 raft_consensus.cc:504] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.544293 8497 leader_election.cc:288] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Requested vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:21.544749 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb8e3aae8c41bba27be3bbe314ff30" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
I20260620 18:30:21.544893 8417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb8e3aae8c41bba27be3bbe314ff30" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c"
I20260620 18:30:21.544975 8139 raft_consensus.cc:3037] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.545112 8417 raft_consensus.cc:3037] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:21.547914 8139 raft_consensus.cc:2454] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:21.548173 8417 raft_consensus.cc:2454] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:21.548313 8209 leader_election.cc:302] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e11a3b34c18243a99b66822123836bcb, eb13bf58548b4a859ce01f94b36a9fd2; no voters:
I20260620 18:30:21.548657 8497 raft_consensus.cc:2781] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:21.548831 8497 raft_consensus.cc:686] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [term 1 LEADER]: Becoming Leader. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Running, Role: LEADER
I20260620 18:30:21.549094 8497 consensus_queue.cc:237] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.550876 8523 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: TxnStatusTablet state changed. Reason: New leader e11a3b34c18243a99b66822123836bcb. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.551275 8523 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:30:21.551951 8535 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:30:21.552170 8535 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:30:21.552388 7981 catalog_manager.cc:5495] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to e11a3b34c18243a99b66822123836bcb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } health_report { overall_health: UNKNOWN } } }
I20260620 18:30:21.581583 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.574297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.0.97.2:41799)
0620 18:30:21.574487 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.574494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.574725 (+ 231us) server_negotiation.cc:407] Connection header received
0620 18:30:21.574989 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.574995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.575123 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.575278 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:21.575703 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.576881 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.578767 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.579406 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.579574 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.580137 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.580838 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.581000 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.581138 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.581227 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:30:21.581667 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.574252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.0.1:52296)
0620 18:30:21.574622 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.574642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.574785 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.575315 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.575320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.575333 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:30:21.575556 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.575567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.577043 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.577049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.578614 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.578628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.579597 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.579605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.579819 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.579893 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.580978 (+ 1085us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.581001 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.581101 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.581259 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260620 18:30:21.586081 8139 raft_consensus.cc:1261] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.586292 8417 raft_consensus.cc:1261] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.587101 8497 consensus_queue.cc:1035] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.587405 8523 consensus_queue.cc:1035] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.599706 8498 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: TxnStatusTablet state changed. Reason: New leader e11a3b34c18243a99b66822123836bcb. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.600046 8498 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.606500 8498 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.606748 8498 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P eb13bf58548b4a859ce01f94b36a9fd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.611617 8523 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.611912 8523 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:30:21.615417 8497 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.615710 8497 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P e11a3b34c18243a99b66822123836bcb: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:30:21.616437 8501 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: TxnStatusTablet state changed. Reason: New leader e11a3b34c18243a99b66822123836bcb. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.616741 8501 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.619678 8501 tablet_replica.cc:439] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } } }
I20260620 18:30:21.619948 8501 tablet_replica.cc:442] T 5eeb8e3aae8c41bba27be3bbe314ff30 P 63b2ba0eeb68469e8aae336407b5713c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:30:21.623023 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.613785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.97.2:41799)
0620 18:30:21.613966 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.613971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.613989 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:21.614048 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.614053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.614139 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.614226 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:21.614761 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.615867 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.621068 (+ 5201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.621708 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.621860 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.622327 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.622424 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.622532 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.622649 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.622719 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:30:21.638520 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.633162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56713 (local address 127.0.97.2:41799)
0620 18:30:21.633353 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.633361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.633478 (+ 117us) server_negotiation.cc:407] Connection header received
0620 18:30:21.633704 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.633709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.633801 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.633894 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:21.634524 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.635266 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.636791 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.637209 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.637362 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.637740 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.637852 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.637947 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.638080 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.638140 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:30:21.638902 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.632656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.97.2:56713)
0620 18:30:21.633394 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:21.633411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:21.633579 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:21.634027 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:21.634031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:21.634041 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:30:21.634394 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.634408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.635388 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.635395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.636652 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:21.636663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.637385 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.637391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:21.637531 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.637575 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:30:21.637956 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:30:21.637968 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:30:21.638064 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:30:21.638186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":722,"threads_started":1}
I20260620 18:30:21.649708 8417 raft_consensus.cc:1261] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.649698 8139 raft_consensus.cc:1261] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:21.650226 8523 consensus_queue.cc:1035] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.650774 8523 consensus_queue.cc:1035] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:21.671190 8505 mvcc.cc:205] Tried to move back new op lower bound from 7298990987871715328 to 7298990986718535680. Current Snapshot: MvccSnapshot[applied={T|T < 7298990987871715328}]
I20260620 18:30:21.771167 7981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:21.771425 7981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:21.775329 7981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Sending DeleteTablet for 3 replicas of tablet 0eb7d81ac51a41178923192b850cab90
I20260620 18:30:21.776280 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7d81ac51a41178923192b850cab90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36926
I20260620 18:30:21.776582 8517 tablet_replica.cc:330] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: stopping tablet replica
I20260620 18:30:21.776800 8517 raft_consensus.cc:2229] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:21.776990 8517 raft_consensus.cc:2258] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.778426 8517 ts_tablet_manager.cc:1905] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.781082 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7d81ac51a41178923192b850cab90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43220
I20260620 18:30:21.781354 8516 tablet_replica.cc:330] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: stopping tablet replica
I20260620 18:30:21.781579 8516 raft_consensus.cc:2229] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:21.781814 8516 raft_consensus.cc:2258] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.782258 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7d81ac51a41178923192b850cab90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 18:30:21.782521 8515 tablet_replica.cc:330] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: stopping tablet replica
I20260620 18:30:21.782717 8515 raft_consensus.cc:2229] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:21.783237 8515 raft_consensus.cc:2258] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:21.784248 8516 ts_tablet_manager.cc:1905] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.788235 8517 ts_tablet_manager.cc:1918] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:21.788472 8517 log.cc:1196] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/0eb7d81ac51a41178923192b850cab90
I20260620 18:30:21.789043 8517 ts_tablet_manager.cc:1939] T 0eb7d81ac51a41178923192b850cab90 P eb13bf58548b4a859ce01f94b36a9fd2: Deleting consensus metadata
I20260620 18:30:21.790319 7966 catalog_manager.cc:4841] TS eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711): tablet 0eb7d81ac51a41178923192b850cab90 (table test-table [id=beacc5d7087c4f78bc28e952ced3ad05]) successfully deleted
I20260620 18:30:21.796561 8515 ts_tablet_manager.cc:1905] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:21.798569 8516 ts_tablet_manager.cc:1918] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:21.802145 8516 log.cc:1196] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/0eb7d81ac51a41178923192b850cab90
I20260620 18:30:21.802649 8516 ts_tablet_manager.cc:1939] T 0eb7d81ac51a41178923192b850cab90 P 63b2ba0eeb68469e8aae336407b5713c: Deleting consensus metadata
I20260620 18:30:21.803920 7966 catalog_manager.cc:4841] TS 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467): tablet 0eb7d81ac51a41178923192b850cab90 (table test-table [id=beacc5d7087c4f78bc28e952ced3ad05]) successfully deleted
I20260620 18:30:21.805258 8515 ts_tablet_manager.cc:1918] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:21.805429 8515 log.cc:1196] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/0eb7d81ac51a41178923192b850cab90
I20260620 18:30:21.805917 8515 ts_tablet_manager.cc:1939] T 0eb7d81ac51a41178923192b850cab90 P e11a3b34c18243a99b66822123836bcb: Deleting consensus metadata
I20260620 18:30:21.807070 7966 catalog_manager.cc:4841] TS e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799): tablet 0eb7d81ac51a41178923192b850cab90 (table test-table [id=beacc5d7087c4f78bc28e952ced3ad05]) successfully deleted
Jun 20 18:30:21 dist-test-slave-mcnm krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:30:21 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:21 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:30:21.912048 8077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:21.884907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.97.62:44251)
0620 18:30:21.885100 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.885108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.900184 (+ 15076us) server_negotiation.cc:407] Connection header received
0620 18:30:21.900397 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.900404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.900490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.900566 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:21.902486 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.903642 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.905408 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.906003 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.906094 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.911796 (+ 5702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59312: BlockingRecv error: recv got EOF from 127.0.0.1:59312 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:30:21 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 18:30:21.933522 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:21.917435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.97.62:44251)
0620 18:30:21.917623 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:21.917628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:21.918174 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:30:21.918337 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:21.918343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:21.918418 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:21.918528 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:21.919962 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.921081 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.922851 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:21.923267 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:21.923368 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:21.928228 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:21.928233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:21.928237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:21.928307 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:21.930901 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:21.931570 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:21.931579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:21.931582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:21.931712 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:21.932096 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:21.932104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:21.932107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:21.932378 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:21.932649 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:21.932936 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:30:21.933156 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:30:21.937305 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:21.938041 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:21.945359 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa (DEFAULT_TABLE table=test-table [id=46dec3c0825047ccb01f819c70353fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.945442 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa (DEFAULT_TABLE table=test-table [id=46dec3c0825047ccb01f819c70353fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.945480 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa (DEFAULT_TABLE table=test-table [id=46dec3c0825047ccb01f819c70353fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:21.945813 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe5695cd8fc44e687c2fe3077ab29fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.945804 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe5695cd8fc44e687c2fe3077ab29fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.945919 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe5695cd8fc44e687c2fe3077ab29fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:21.952495 8494 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Bootstrap starting.
I20260620 18:30:21.952782 8493 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Bootstrap starting.
I20260620 18:30:21.953928 8493 tablet_bootstrap.cc:654] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.954125 8494 tablet_bootstrap.cc:654] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.955938 8493 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: No bootstrap required, opened a new log
I20260620 18:30:21.956099 8493 ts_tablet_manager.cc:1397] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:30:21.956336 8562 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Bootstrap starting.
I20260620 18:30:21.956856 8493 raft_consensus.cc:348] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.957136 8493 raft_consensus.cc:374] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.957243 8493 raft_consensus.cc:729] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.957647 8493 consensus_queue.cc:260] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.958215 8562 tablet_bootstrap.cc:654] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:21.961618 8493 ts_tablet_manager.cc:1428] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:30:21.966024 8562 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: No bootstrap required, opened a new log
I20260620 18:30:21.966238 8562 ts_tablet_manager.cc:1397] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260620 18:30:21.966692 8494 tablet_bootstrap.cc:492] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: No bootstrap required, opened a new log
I20260620 18:30:21.966861 8494 ts_tablet_manager.cc:1397] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20260620 18:30:21.967031 8562 raft_consensus.cc:348] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.967268 8562 raft_consensus.cc:374] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.967367 8562 raft_consensus.cc:729] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.967666 8562 consensus_queue.cc:260] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.967675 8494 raft_consensus.cc:348] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.967947 8494 raft_consensus.cc:374] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:21.968043 8494 raft_consensus.cc:729] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Initialized, Role: FOLLOWER
I20260620 18:30:21.968325 8494 consensus_queue.cc:260] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.968701 8562 ts_tablet_manager.cc:1428] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:30:21.969496 8494 ts_tablet_manager.cc:1428] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:21.994182 8547 raft_consensus.cc:482] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:21.994411 8547 raft_consensus.cc:504] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:21.995433 8547 leader_election.cc:288] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:21.995941 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe5695cd8fc44e687c2fe3077ab29fa" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c" is_pre_election: true
I20260620 18:30:21.995910 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe5695cd8fc44e687c2fe3077ab29fa" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" is_pre_election: true
I20260620 18:30:21.996186 8416 raft_consensus.cc:2454] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.996187 8139 raft_consensus.cc:2454] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 0.
I20260620 18:30:21.996665 8209 leader_election.cc:302] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, e11a3b34c18243a99b66822123836bcb; no voters:
I20260620 18:30:21.997002 8547 raft_consensus.cc:2781] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:21.997140 8547 raft_consensus.cc:482] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:21.997258 8547 raft_consensus.cc:3037] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.000975 8547 raft_consensus.cc:504] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.001729 8547 leader_election.cc:288] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Requested vote from peers eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:22.002158 8139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe5695cd8fc44e687c2fe3077ab29fa" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb13bf58548b4a859ce01f94b36a9fd2"
I20260620 18:30:22.002302 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe5695cd8fc44e687c2fe3077ab29fa" candidate_uuid: "e11a3b34c18243a99b66822123836bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c"
I20260620 18:30:22.002377 8139 raft_consensus.cc:3037] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.002518 8416 raft_consensus.cc:3037] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.005317 8416 raft_consensus.cc:2454] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:22.005317 8139 raft_consensus.cc:2454] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11a3b34c18243a99b66822123836bcb in term 1.
I20260620 18:30:22.005702 8209 leader_election.cc:302] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, e11a3b34c18243a99b66822123836bcb; no voters:
I20260620 18:30:22.006049 8547 raft_consensus.cc:2781] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:22.006249 8547 raft_consensus.cc:686] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 1 LEADER]: Becoming Leader. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Running, Role: LEADER
I20260620 18:30:22.006554 8547 consensus_queue.cc:237] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.009573 7982 catalog_manager.cc:5495] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to e11a3b34c18243a99b66822123836bcb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e11a3b34c18243a99b66822123836bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } health_report { overall_health: UNKNOWN } } }
I20260620 18:30:22.035154 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.030198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.0.97.2:41799)
0620 18:30:22.030400 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.030406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.030494 (+ 88us) server_negotiation.cc:407] Connection header received
0620 18:30:22.030715 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.030728 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.030797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.030894 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:22.031260 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.032050 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.033368 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.033799 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.033935 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.034361 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.034492 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.034613 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.034771 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.034850 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:30:22.037869 8416 raft_consensus.cc:1261] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:22.037920 8139 raft_consensus.cc:1261] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Refusing update from remote peer e11a3b34c18243a99b66822123836bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:22.038486 8547 consensus_queue.cc:1035] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:22.038782 8543 consensus_queue.cc:1035] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:22.051267 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:22.051494 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:22.056563 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Sending DeleteTablet for 3 replicas of tablet 4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.057559 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36926
I20260620 18:30:22.057739 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 18:30:22.057837 8517 tablet_replica.cc:330] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: stopping tablet replica
I20260620 18:30:22.058035 8515 tablet_replica.cc:330] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: stopping tablet replica
I20260620 18:30:22.058049 8517 raft_consensus.cc:2229] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:22.058199 8515 raft_consensus.cc:2229] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:22.058266 8517 raft_consensus.cc:2258] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.058516 8515 raft_consensus.cc:2258] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.059417 8515 ts_tablet_manager.cc:1905] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.059862 8517 ts_tablet_manager.cc:1905] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.060516 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43220
I20260620 18:30:22.063664 8397 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43220
I20260620 18:30:22.063674 8516 tablet_replica.cc:330] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: stopping tablet replica
I20260620 18:30:22.063970 8516 raft_consensus.cc:2229] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:22.064066 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe5695cd8fc44e687c2fe3077ab29fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 18:30:22.064182 8516 raft_consensus.cc:2258] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.065483 8516 ts_tablet_manager.cc:1905] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.067131 8515 ts_tablet_manager.cc:1918] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:22.067126 8517 ts_tablet_manager.cc:1918] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:22.067241 8517 log.cc:1196] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.067250 8515 log.cc:1196] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.067700 8517 ts_tablet_manager.cc:1939] T 4fe5695cd8fc44e687c2fe3077ab29fa P eb13bf58548b4a859ce01f94b36a9fd2: Deleting consensus metadata
I20260620 18:30:22.067713 8515 ts_tablet_manager.cc:1939] T 4fe5695cd8fc44e687c2fe3077ab29fa P e11a3b34c18243a99b66822123836bcb: Deleting consensus metadata
I20260620 18:30:22.068692 7966 catalog_manager.cc:4841] TS eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711): tablet 4fe5695cd8fc44e687c2fe3077ab29fa (table test-table [id=46dec3c0825047ccb01f819c70353fd7]) successfully deleted
I20260620 18:30:22.068733 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:30:22.069288 7966 catalog_manager.cc:4841] TS e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799): tablet 4fe5695cd8fc44e687c2fe3077ab29fa (table test-table [id=46dec3c0825047ccb01f819c70353fd7]) successfully deleted
W20260620 18:30:22.069380 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 18:30:22.069759 7966 catalog_manager.cc:4808] TS e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799): delete failed for tablet 4fe5695cd8fc44e687c2fe3077ab29fa because the tablet was not found. No further retry: Not found: Tablet not found: 4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.075255 8516 ts_tablet_manager.cc:1918] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:22.075403 8516 log.cc:1196] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.075881 8516 ts_tablet_manager.cc:1939] T 4fe5695cd8fc44e687c2fe3077ab29fa P 63b2ba0eeb68469e8aae336407b5713c: Deleting consensus metadata
I20260620 18:30:22.076445 8119 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f (DEFAULT_TABLE table=test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:22.076879 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2eee6636a472eb5bea5b39921ab3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:22.078008 8397 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f (DEFAULT_TABLE table=test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:22.078104 8259 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f (DEFAULT_TABLE table=test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:22.078356 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2eee6636a472eb5bea5b39921ab3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:22.078508 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2eee6636a472eb5bea5b39921ab3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:22.083117 8493 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Bootstrap starting.
I20260620 18:30:22.084342 8493 tablet_bootstrap.cc:654] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:22.085646 7966 catalog_manager.cc:4841] TS 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467): tablet 4fe5695cd8fc44e687c2fe3077ab29fa (table test-table [id=46dec3c0825047ccb01f819c70353fd7]) successfully deleted
W20260620 18:30:22.085943 7966 catalog_manager.cc:4808] TS 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467): delete failed for tablet 4fe5695cd8fc44e687c2fe3077ab29fa because the tablet was not found. No further retry: Not found: Tablet not found: 4fe5695cd8fc44e687c2fe3077ab29fa
I20260620 18:30:22.086122 8493 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: No bootstrap required, opened a new log
I20260620 18:30:22.086283 8493 ts_tablet_manager.cc:1397] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:22.086632 8562 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Bootstrap starting.
I20260620 18:30:22.087015 8493 raft_consensus.cc:348] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.087246 8493 raft_consensus.cc:374] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:22.087325 8493 raft_consensus.cc:729] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Initialized, Role: FOLLOWER
I20260620 18:30:22.087664 8493 consensus_queue.cc:260] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.088387 8562 tablet_bootstrap.cc:654] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:22.088758 8494 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Bootstrap starting.
I20260620 18:30:22.090250 8494 tablet_bootstrap.cc:654] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:22.090781 8493 ts_tablet_manager.cc:1428] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260620 18:30:22.094851 8562 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: No bootstrap required, opened a new log
I20260620 18:30:22.095041 8562 ts_tablet_manager.cc:1397] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 18:30:22.095255 8494 tablet_bootstrap.cc:492] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: No bootstrap required, opened a new log
I20260620 18:30:22.095423 8494 ts_tablet_manager.cc:1397] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260620 18:30:22.095849 8562 raft_consensus.cc:348] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.096069 8562 raft_consensus.cc:374] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:22.096156 8562 raft_consensus.cc:729] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11a3b34c18243a99b66822123836bcb, State: Initialized, Role: FOLLOWER
I20260620 18:30:22.096151 8494 raft_consensus.cc:348] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.096419 8494 raft_consensus.cc:374] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:22.096498 8494 raft_consensus.cc:729] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2ba0eeb68469e8aae336407b5713c, State: Initialized, Role: FOLLOWER
I20260620 18:30:22.096406 8562 consensus_queue.cc:260] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.096750 8494 consensus_queue.cc:260] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.096889 8562 ts_tablet_manager.cc:1428] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:30:22.098508 8494 ts_tablet_manager.cc:1428] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:22.294523 8498 raft_consensus.cc:482] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:22.294760 8498 raft_consensus.cc:504] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.296321 8498 leader_election.cc:288] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:22.304114 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.297195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42787 (local address 127.0.97.2:41799)
0620 18:30:22.297391 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.297399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.297421 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:30:22.297632 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.297639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.297745 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.297820 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:22.298590 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.299657 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.301803 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.303104 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.303254 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.303348 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.303692 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.303828 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:30:22.304822 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.297327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42467 (local address 127.0.97.1:37673)
0620 18:30:22.297879 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:22.297897 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:22.298018 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:22.298820 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:22.298824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:22.298849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:22.299202 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.299213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.300624 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.300629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.302339 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.302349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.304015 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.304022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.304276 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.304305 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:30:22.304383 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:30:22.304508 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":464,"threads_started":1}
I20260620 18:30:22.304832 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.297632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37673 (local address 127.0.97.3:42467)
0620 18:30:22.298525 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.298531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.298548 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:22.298613 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.298618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.298683 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.298790 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:22.299365 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.300489 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.302543 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.303737 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.303998 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.304112 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.304424 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.304552 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":578,"threads_started":1}
I20260620 18:30:22.305266 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.296540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.97.1:42787)
0620 18:30:22.297276 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:22.297295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:22.297425 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:22.298074 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:22.298082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:22.298119 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:22.298424 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.298433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.299786 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.299792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.301625 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.301639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.303282 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.303290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.303504 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.303560 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:30:22.303656 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:30:22.303780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":29,"thread_start_us":403,"threads_started":1}
I20260620 18:30:22.305626 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2eee6636a472eb5bea5b39921ab3f" candidate_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c" is_pre_election: true
I20260620 18:30:22.305862 8416 raft_consensus.cc:2454] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb13bf58548b4a859ce01f94b36a9fd2 in term 0.
I20260620 18:30:22.305948 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2eee6636a472eb5bea5b39921ab3f" candidate_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11a3b34c18243a99b66822123836bcb" is_pre_election: true
I20260620 18:30:22.306195 8283 raft_consensus.cc:2454] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb13bf58548b4a859ce01f94b36a9fd2 in term 0.
I20260620 18:30:22.306324 8069 leader_election.cc:302] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, eb13bf58548b4a859ce01f94b36a9fd2; no voters:
I20260620 18:30:22.306758 8498 raft_consensus.cc:2781] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:22.306896 8498 raft_consensus.cc:482] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:22.307013 8498 raft_consensus.cc:3037] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.310073 8498 raft_consensus.cc:504] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.310724 8498 leader_election.cc:288] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [CANDIDATE]: Term 1 election: Requested vote from peers e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799), 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467)
I20260620 18:30:22.311254 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2eee6636a472eb5bea5b39921ab3f" candidate_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2ba0eeb68469e8aae336407b5713c"
I20260620 18:30:22.311337 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2eee6636a472eb5bea5b39921ab3f" candidate_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11a3b34c18243a99b66822123836bcb"
I20260620 18:30:22.311447 8416 raft_consensus.cc:3037] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.311561 8283 raft_consensus.cc:3037] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:22.313717 8416 raft_consensus.cc:2454] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb13bf58548b4a859ce01f94b36a9fd2 in term 1.
I20260620 18:30:22.314080 8069 leader_election.cc:302] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2ba0eeb68469e8aae336407b5713c, eb13bf58548b4a859ce01f94b36a9fd2; no voters:
I20260620 18:30:22.314348 8498 raft_consensus.cc:2781] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:22.314491 8283 raft_consensus.cc:2454] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb13bf58548b4a859ce01f94b36a9fd2 in term 1.
I20260620 18:30:22.314863 8498 raft_consensus.cc:686] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 LEADER]: Becoming Leader. State: Replica: eb13bf58548b4a859ce01f94b36a9fd2, State: Running, Role: LEADER
I20260620 18:30:22.315236 8498 consensus_queue.cc:237] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } }
I20260620 18:30:22.318616 7982 catalog_manager.cc:5495] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb13bf58548b4a859ce01f94b36a9fd2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 } health_report { overall_health: UNKNOWN } } }
I20260620 18:30:22.343212 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.336640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.97.1:42711)
0620 18:30:22.337234 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.337238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.337255 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:22.337331 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.337334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.337402 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.337589 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:22.338012 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.338924 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.340823 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.341284 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.341397 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.341943 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.342533 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.342648 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.342788 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.342850 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":362,"threads_started":1}
I20260620 18:30:22.359164 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.353271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57419 (local address 127.0.97.2:41799)
0620 18:30:22.353475 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.353483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.353859 (+ 376us) server_negotiation.cc:407] Connection header received
0620 18:30:22.354024 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.354027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.354087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.354162 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:22.354712 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.355425 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.357474 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.357917 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.358053 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.358543 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.358634 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.358703 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.358855 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.358910 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:30:22.359395 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.353033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41799 (local address 127.0.97.1:57419)
0620 18:30:22.353740 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:22.353760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:22.353900 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:22.354205 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:22.354210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:22.354223 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:30:22.354554 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.354565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.355581 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.355587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.357345 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.357361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.358132 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.358139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.358321 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.358385 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.358732 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.358743 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:30:22.358824 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:30:22.358958 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":357,"threads_started":1}
I20260620 18:30:22.367982 8416 raft_consensus.cc:1261] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Refusing update from remote peer eb13bf58548b4a859ce01f94b36a9fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:22.368162 8283 raft_consensus.cc:1261] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Refusing update from remote peer eb13bf58548b4a859ce01f94b36a9fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:22.368817 8575 consensus_queue.cc:1035] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11a3b34c18243a99b66822123836bcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:22.369470 8498 consensus_queue.cc:1035] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b2ba0eeb68469e8aae336407b5713c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:22.415808 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.399167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32969 (local address 127.0.97.1:42711)
0620 18:30:22.399352 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.399361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.408347 (+ 8986us) server_negotiation.cc:407] Connection header received
0620 18:30:22.408524 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.408530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.408606 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.408695 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:22.409379 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.410505 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.413252 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.413913 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.414030 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.414605 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.414733 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.414800 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.415395 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.415483 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:30:22.416747 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.398906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42711 (local address 127.0.97.2:32969)
0620 18:30:22.408211 (+ 9305us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:22.408247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:22.408390 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:22.408744 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:22.408748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:22.408761 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:30:22.409169 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.409183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.410895 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.410901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.413115 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:22.413130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.414101 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.414113 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:22.414358 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.414409 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:30:22.415226 (+ 817us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:30:22.415248 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:30:22.416212 (+ 964us) client_negotiation.cc:240] Negotiation successful
0620 18:30:22.416357 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9121,"thread_start_us":8978,"threads_started":1}
I20260620 18:30:22.453362 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:22.453624 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:22.457093 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6898f32fb9410ab9b0a7a4c77428ee: Sending DeleteTablet for 3 replicas of tablet b5c2eee6636a472eb5bea5b39921ab3f
I20260620 18:30:22.457969 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36926
I20260620 18:30:22.458097 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 18:30:22.458375 8515 tablet_replica.cc:330] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: stopping tablet replica
I20260620 18:30:22.458581 8515 raft_consensus.cc:2229] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:22.458699 8517 tablet_replica.cc:330] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: stopping tablet replica
I20260620 18:30:22.458768 8515 raft_consensus.cc:2258] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.458909 8517 raft_consensus.cc:2229] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:22.459425 8517 raft_consensus.cc:2258] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.460317 8515 ts_tablet_manager.cc:1905] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.460872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8053
I20260620 18:30:22.464989 8517 ts_tablet_manager.cc:1905] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.466902 8397 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2eee6636a472eb5bea5b39921ab3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:22 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43220
I20260620 18:30:22.467180 8516 tablet_replica.cc:330] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: stopping tablet replica
I20260620 18:30:22.467308 8515 ts_tablet_manager.cc:1918] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:22.467373 8516 raft_consensus.cc:2229] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:22.467450 8515 log.cc:1196] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/b5c2eee6636a472eb5bea5b39921ab3f
I20260620 18:30:22.467612 8516 raft_consensus.cc:2258] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:22.467908 8515 ts_tablet_manager.cc:1939] T b5c2eee6636a472eb5bea5b39921ab3f P e11a3b34c18243a99b66822123836bcb: Deleting consensus metadata
I20260620 18:30:22.468938 8516 ts_tablet_manager.cc:1905] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:22.473452 8517 ts_tablet_manager.cc:1918] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:22.473672 8517 log.cc:1196] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/b5c2eee6636a472eb5bea5b39921ab3f
I20260620 18:30:22.474104 8517 ts_tablet_manager.cc:1939] T b5c2eee6636a472eb5bea5b39921ab3f P eb13bf58548b4a859ce01f94b36a9fd2: Deleting consensus metadata
I20260620 18:30:22.475354 8516 ts_tablet_manager.cc:1918] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:30:22.475500 8516 log.cc:1196] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/b5c2eee6636a472eb5bea5b39921ab3f
I20260620 18:30:22.475916 8516 ts_tablet_manager.cc:1939] T b5c2eee6636a472eb5bea5b39921ab3f P 63b2ba0eeb68469e8aae336407b5713c: Deleting consensus metadata
I20260620 18:30:22.476701 7966 catalog_manager.cc:4841] TS eb13bf58548b4a859ce01f94b36a9fd2 (127.0.97.1:42711): tablet b5c2eee6636a472eb5bea5b39921ab3f (table test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]) successfully deleted
I20260620 18:30:22.476977 7966 catalog_manager.cc:4841] TS 63b2ba0eeb68469e8aae336407b5713c (127.0.97.3:42467): tablet b5c2eee6636a472eb5bea5b39921ab3f (table test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]) successfully deleted
I20260620 18:30:22.477559 7966 catalog_manager.cc:4841] TS e11a3b34c18243a99b66822123836bcb (127.0.97.2:41799): tablet b5c2eee6636a472eb5bea5b39921ab3f (table test-table [id=ac4f27d2e3d94a3db7ada04e779c56da]) successfully deleted
Jun 20 18:30:22 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 18:30:22.495327 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.462351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.97.1:42711)
0620 18:30:22.462543 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.462550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.474904 (+ 12354us) server_negotiation.cc:407] Connection header received
0620 18:30:22.475356 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.475362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.475458 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.475572 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:22.477944 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.479071 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.480997 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.481690 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.481821 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.487387 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:22.487401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:22.487423 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:22.487471 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:22.491905 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.492713 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.492722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.492725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.492934 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.493364 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.493370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.493377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.493834 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:22.494189 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.494477 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.494701 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:30:22.496315 8184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:22.603444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8053
I20260620 18:30:22.623333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8193
Jun 20 18:30:22 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 18:30:22.637596 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.624099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.0.97.2:41799)
0620 18:30:22.624387 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.624393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.624494 (+ 101us) server_negotiation.cc:407] Connection header received
0620 18:30:22.624707 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.624712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.624784 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.624955 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:22.626114 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.626782 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.628096 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.628514 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.628645 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.632586 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:22.632598 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:22.632615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:22.632649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:22.635284 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.635917 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.635923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.635926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.636064 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.636363 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.636367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.636369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.636726 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:22.636984 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.637181 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.637357 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 18:30:22.638249 8323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:22.785140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8193
I20260620 18:30:22.810109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8332
Jun 20 18:30:22 dist-test-slave-mcnm krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 18:30:22.824776 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:22.810873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.97.3:42467)
0620 18:30:22.811375 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:22.811379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:22.811395 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:30:22.811448 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:22.811452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:22.811548 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:22.811645 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:22.812805 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.813500 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.814653 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:22.815036 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:22.815122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:22.819500 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:22.819551 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:22.819574 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:22.819619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:22.822402 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.823037 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.823043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.823046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.823158 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:22.823577 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:22.823580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:22.823582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:22.823888 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:22.824108 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:22.824368 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:30:22.824520 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":300,"threads_started":1}
I20260620 18:30:22.825397 8462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:22.945744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8332
I20260620 18:30:22.967851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 7950
I20260620 18:30:22.969000 8022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:23.082392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 7950
2026-06-20T18:30:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5566 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:23 dist-test-slave-mcnm krb5kdc[8595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:23 dist-test-slave-mcnm krb5kdc[8595](info): set up 2 sockets
Jun 20 18:30:23 dist-test-slave-mcnm krb5kdc[8595](info): commencing operation
krb5kdc: starting...
W20260620 18:30:24.196740 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:24 dist-test-slave-mcnm krb5kdc[8595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:30:24.405416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33551
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:24.718843 8608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:24.719132 8608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:24.719302 8608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:24.726501 8608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:24.726606 8608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:24.726679 8608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:24.726765 8608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:24.726853 8608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:24.736665 8608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:24.737946 8608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:24.739287 8608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:24.751319 8614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:24.751394 8613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:24.751428 8616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:24.752540 8608 server_base.cc:1034] running on GCE node
I20260620 18:30:24.753396 8608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:24.755504 8608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:24.756923 8608 hybrid_clock.cc:648] HybridClock initialized: now 1781980224756835 us; error 86 us; skew 500 ppm
I20260620 18:30:24.757365 8608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:30:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1747 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:24 dist-test-slave-mcnm krb5kdc[8622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:24 dist-test-slave-mcnm krb5kdc[8622](info): set up 2 sockets
Jun 20 18:30:24 dist-test-slave-mcnm krb5kdc[8622](info): commencing operation
krb5kdc: starting...
W20260620 18:30:25.936651 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:26 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:30:26.157631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43487
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:26.473742 8635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:26.474018 8635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:26.474249 8635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:26.481320 8635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:26.481426 8635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:26.481532 8635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:26.481628 8635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:26.481714 8635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:26.491266 8635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:26.492606 8635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:26.493892 8635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:26.505393 8643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:26.505461 8640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:26.505522 8641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:26.506279 8635 server_base.cc:1034] running on GCE node
I20260620 18:30:26.506939 8635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:26.508327 8635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:26.509564 8635 hybrid_clock.cc:648] HybridClock initialized: now 1781980226509513 us; error 39 us; skew 500 ppm
I20260620 18:30:26.509903 8635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:26 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:26.513114 8635 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:30:26.514533 8635 webserver.cc:460] Webserver started at http://127.0.97.62:34657/ using document root <none> and password file <none>
I20260620 18:30:26.515208 8635 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:26.515352 8635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:26.515718 8635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:26.518543 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "32aa04146ea544d4b2b2b21bbba29f9e"
format_stamp: "Formatted at 2026-06-20 18:30:26 on dist-test-slave-mcnm"
server_key: "677af09bd4594567d1e5d34d4e4e9143"
server_key_iv: "d9ae018cf2e5d73f6d286f42c9895e18"
server_key_version: "encryptionkey@0"
I20260620 18:30:26.519300 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "32aa04146ea544d4b2b2b21bbba29f9e"
format_stamp: "Formatted at 2026-06-20 18:30:26 on dist-test-slave-mcnm"
server_key: "677af09bd4594567d1e5d34d4e4e9143"
server_key_iv: "d9ae018cf2e5d73f6d286f42c9895e18"
server_key_version: "encryptionkey@0"
I20260620 18:30:26.524304 8635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:30:26.527745 8650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:26.529129 8635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:30:26.529309 8635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "32aa04146ea544d4b2b2b21bbba29f9e"
format_stamp: "Formatted at 2026-06-20 18:30:26 on dist-test-slave-mcnm"
server_key: "677af09bd4594567d1e5d34d4e4e9143"
server_key_iv: "d9ae018cf2e5d73f6d286f42c9895e18"
server_key_version: "encryptionkey@0"
I20260620 18:30:26.529506 8635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:26.543431 8635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:26.547760 8635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:26.548043 8635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:26.566745 8635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43487
I20260620 18:30:26.566799 8701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43487 every 8 connection(s)
I20260620 18:30:26.568141 8635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:26.571920 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:26.577194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8635
I20260620 18:30:26.577392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:26.578176 388 external_mini_cluster.cc:1428] Setting key 4d50dab1fe736f4dfbcff9676464bb69
I20260620 18:30:26.583081 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Bootstrap starting.
I20260620 18:30:26.586648 8702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:26.587940 8702 log.cc:824] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:26.591256 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: No bootstrap required, opened a new log
Jun 20 18:30:26 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:26.601675 8702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:26.602114 8702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:26.602222 8702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa04146ea544d4b2b2b21bbba29f9e, State: Initialized, Role: FOLLOWER
I20260620 18:30:26.602869 8702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:26.603132 8702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:26.603209 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:26.580260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.97.62:43487)
0620 18:30:26.581135 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:26.581149 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:26.581190 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:30:26.582801 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:26.582822 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:26.583242 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:26.583620 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:26.585386 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:26.586866 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:26.588703 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:26.589592 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:26.589735 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:26.594598 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:26.594614 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:26.594633 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:26.594679 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:26.598655 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:26.599405 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:26.599421 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:26.599431 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:26.599674 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:26.600048 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:26.600053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:26.600056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:26.600772 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:26.601063 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:26.601458 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 18:30:26.601868 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":431,"threads_started":1}
I20260620 18:30:26.603287 8702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:26.603470 8702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:26.606144 8702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:26.606586 8702 leader_election.cc:302] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32aa04146ea544d4b2b2b21bbba29f9e; no voters:
I20260620 18:30:26.607123 8702 leader_election.cc:288] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:26.607266 8707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:26.608057 8707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 LEADER]: Becoming Leader. State: Replica: 32aa04146ea544d4b2b2b21bbba29f9e, State: Running, Role: LEADER
I20260620 18:30:26.608616 8707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:26.609042 8702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:26.611272 8709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32aa04146ea544d4b2b2b21bbba29f9e. Latest consensus state: current_term: 1 leader_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } } }
I20260620 18:30:26.611696 8709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:26.611301 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } } }
I20260620 18:30:26.613891 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:26.616539 8712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:26.621061 8712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:26.629105 8712 catalog_manager.cc:1347] Generated new cluster ID: 041abe2e9a344c7c859d2fbcfe1d87a4
I20260620 18:30:26.629199 8712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:26.651011 8712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:26.652818 8712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:26.668658 8712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Generated new TSK 0
I20260620 18:30:26.669734 8712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:30:26.732867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:27.058683 8728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:27.058947 8728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:27.059094 8728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:27.066213 8728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:27.066322 8728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:27.066505 8728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:27.076195 8728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:27.077471 8728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:27.078763 8728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:27.091900 8734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:27.091974 8733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:27.092000 8736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:27.094259 8728 server_base.cc:1034] running on GCE node
I20260620 18:30:27.094903 8728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:27.095912 8728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:27.097159 8728 hybrid_clock.cc:648] HybridClock initialized: now 1781980227097116 us; error 47 us; skew 500 ppm
I20260620 18:30:27.097517 8728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:27 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:27.106037 8728 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:30:27.107764 8728 webserver.cc:460] Webserver started at http://127.0.97.1:36115/ using document root <none> and password file <none>
I20260620 18:30:27.108503 8728 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:27.108611 8728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:27.108939 8728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:27.112113 8728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "dfca6bdd8cbf45d6bf42a20e49098030"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "65efeb168c0c89830117945b2e6d9a56"
server_key_iv: "35db0349bf3ec2c2104abd392ecd54d9"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.112876 8728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dfca6bdd8cbf45d6bf42a20e49098030"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "65efeb168c0c89830117945b2e6d9a56"
server_key_iv: "35db0349bf3ec2c2104abd392ecd54d9"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.118330 8728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:27.121809 8743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.123147 8728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:27.123337 8728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "dfca6bdd8cbf45d6bf42a20e49098030"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "65efeb168c0c89830117945b2e6d9a56"
server_key_iv: "35db0349bf3ec2c2104abd392ecd54d9"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.123617 8728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:27.148761 8728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:27.152864 8728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:27.153160 8728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:27.154096 8728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:27.155436 8728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:27.155560 8728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.155704 8728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:27.155776 8728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.189133 8728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34137
I20260620 18:30:27.189219 8855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34137 every 8 connection(s)
I20260620 18:30:27.190374 8728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:27.196087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8728
I20260620 18:30:27.196286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:27.196934 388 external_mini_cluster.cc:1428] Setting key 4fc5c13ca626a3a92b3dbe710447b07c
Jun 20 18:30:27 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:27.211947 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:27.192932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44383 (local address 127.0.97.62:43487)
0620 18:30:27.193394 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:27.193399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:27.194255 (+ 856us) server_negotiation.cc:407] Connection header received
0620 18:30:27.196188 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:27.196193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:27.196280 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:27.196397 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:27.198018 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.198890 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.200486 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.200873 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.201002 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:27.204802 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:27.204811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:27.204816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:27.204869 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:27.208424 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:27.209297 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:27.209305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:27.209312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:27.209453 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:27.209881 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:27.209889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:27.209896 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:27.210251 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:27.210409 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:27.211193 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 18:30:27.211435 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260620 18:30:27.212639 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:27.193354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:44383)
0620 18:30:27.194086 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:27.194128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:27.195920 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:27.196626 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:27.196635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:27.197195 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:27.197857 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:27.197876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.199050 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.199055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:27.200329 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:27.200350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.201110 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.201120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:27.201345 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:27.202014 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:27.202035 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:27.204589 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:27.208553 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:27.208560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:27.208573 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:27.209166 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:27.209555 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:27.209558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:27.209560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:27.209771 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:27.210451 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:27.210457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:27.210781 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 18:30:27.211039 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 18:30:27.211430 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":277,"threads_started":1}
I20260620 18:30:27.214233 8856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43487
I20260620 18:30:27.214563 8856 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:27.215468 8856 heartbeater.cc:511] Master 127.0.97.62:43487 requested a full tablet report, sending...
I20260620 18:30:27.218124 8667 ts_manager.cc:194] Registered new tserver with Master: dfca6bdd8cbf45d6bf42a20e49098030 (127.0.97.1:34137)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:30:27.219759 8667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:30:27.246747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:27.571682 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:27.572003 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:27.572162 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:27.583496 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:27.583649 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:27.583829 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:27.595443 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:27.596709 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:27.597993 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:27.611150 8867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:27.611114 8868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:27.611114 8870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:27.612150 8862 server_base.cc:1034] running on GCE node
I20260620 18:30:27.612752 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:27.613762 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:27.614984 8862 hybrid_clock.cc:648] HybridClock initialized: now 1781980227614941 us; error 37 us; skew 500 ppm
I20260620 18:30:27.615324 8862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:27 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:27.618324 8862 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:30:27.619729 8862 webserver.cc:460] Webserver started at http://127.0.97.2:35105/ using document root <none> and password file <none>
I20260620 18:30:27.620398 8862 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:27.620532 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:27.620828 8862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:27.623811 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "acacd94d66b44f6283dc7c103b2ef878"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "da1b3cee32e6ac25427e09a85f163227"
server_key_iv: "70c88a9002d3de71a773cc4983187a3c"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.624583 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "acacd94d66b44f6283dc7c103b2ef878"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "da1b3cee32e6ac25427e09a85f163227"
server_key_iv: "70c88a9002d3de71a773cc4983187a3c"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.629712 8862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:27.633219 8877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.634543 8862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:30:27.634666 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "acacd94d66b44f6283dc7c103b2ef878"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "da1b3cee32e6ac25427e09a85f163227"
server_key_iv: "70c88a9002d3de71a773cc4983187a3c"
server_key_version: "encryptionkey@0"
I20260620 18:30:27.634838 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:27.647626 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:27.651492 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:27.651800 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:27.652619 8862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:27.653792 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:27.653861 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.653939 8862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:27.653993 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:27.685307 8862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44949
I20260620 18:30:27.685451 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44949 every 8 connection(s)
I20260620 18:30:27.686558 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:27.686985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8862
I20260620 18:30:27.687175 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:27.687887 388 external_mini_cluster.cc:1428] Setting key f03116c418cc860f68542382753c180d
Jun 20 18:30:27 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:27.706918 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:27.689142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55905 (local address 127.0.97.62:43487)
0620 18:30:27.689337 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:27.689345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:27.690618 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 18:30:27.692572 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:27.692580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:27.692676 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:27.692796 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:27.694356 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.695684 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.697249 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.697696 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.697829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:27.701455 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:27.701460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:27.701463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:27.701495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:27.704072 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:27.704739 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:27.704746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:27.704749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:27.704876 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:27.705275 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:27.705290 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:27.705294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:27.705538 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:27.705696 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:27.706352 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 18:30:27.706577 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:30:27.707870 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:27.689661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:55905)
0620 18:30:27.690438 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:27.690490 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:27.692273 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:27.693032 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:27.693041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:27.693600 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:27.694156 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:27.694173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.695838 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.695843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:27.697095 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:27.697107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:27.697928 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:27.697935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:27.698149 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:27.698812 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:27.698835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:27.701224 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:27.704210 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:27.704219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:27.704234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:27.704625 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:27.704980 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:27.704987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:27.704990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:27.705171 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:27.705660 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:27.705669 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:27.705986 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 18:30:27.706258 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:30:27.706757 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:30:27.709424 8990 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43487
I20260620 18:30:27.709734 8990 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:27.710649 8990 heartbeater.cc:511] Master 127.0.97.62:43487 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 18:30:27.712273 8667 ts_manager.cc:194] Registered new tserver with Master: acacd94d66b44f6283dc7c103b2ef878 (127.0.97.2:44949)
I20260620 18:30:27.713049 8667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:30:27.738960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:28.057883 8995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:28.058140 8995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:28.058290 8995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:28.065271 8995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:28.065376 8995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:28.065538 8995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:28.075057 8995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:28.076315 8995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:28.077602 8995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:28.091238 9000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:28.091212 9003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:28.091177 9001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:28.091917 8995 server_base.cc:1034] running on GCE node
I20260620 18:30:28.092495 8995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:28.093371 8995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:28.094585 8995 hybrid_clock.cc:648] HybridClock initialized: now 1781980228094549 us; error 56 us; skew 500 ppm
I20260620 18:30:28.094878 8995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:28 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:28.098008 8995 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:30:28.099372 8995 webserver.cc:460] Webserver started at http://127.0.97.3:32953/ using document root <none> and password file <none>
I20260620 18:30:28.100047 8995 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:28.100180 8995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:28.100493 8995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:28.103276 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "967e81844f4f418ba8974df3423fa0cd"
format_stamp: "Formatted at 2026-06-20 18:30:28 on dist-test-slave-mcnm"
server_key: "eaa126d8b131292549f04ce4a2649702"
server_key_iv: "3a6cbd572fc9ed89fabd2b4017cb4794"
server_key_version: "encryptionkey@0"
I20260620 18:30:28.104102 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "967e81844f4f418ba8974df3423fa0cd"
format_stamp: "Formatted at 2026-06-20 18:30:28 on dist-test-slave-mcnm"
server_key: "eaa126d8b131292549f04ce4a2649702"
server_key_iv: "3a6cbd572fc9ed89fabd2b4017cb4794"
server_key_version: "encryptionkey@0"
I20260620 18:30:28.109119 8995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:30:28.112478 9010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:28.113715 8995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:30:28.113879 8995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "967e81844f4f418ba8974df3423fa0cd"
format_stamp: "Formatted at 2026-06-20 18:30:28 on dist-test-slave-mcnm"
server_key: "eaa126d8b131292549f04ce4a2649702"
server_key_iv: "3a6cbd572fc9ed89fabd2b4017cb4794"
server_key_version: "encryptionkey@0"
I20260620 18:30:28.114074 8995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:28.150353 8995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:28.154479 8995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:28.154758 8995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:28.155752 8995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:28.157058 8995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:28.157155 8995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:28.157306 8995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:28.157375 8995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:28.190711 8995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38231
I20260620 18:30:28.190747 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38231 every 8 connection(s)
I20260620 18:30:28.192221 8995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:28.199625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 8995
I20260620 18:30:28.199781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:28.200192 388 external_mini_cluster.cc:1428] Setting key c08b0cf29b1b030f63da66ce884ebd28
Jun 20 18:30:28 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:28.211457 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:28.194722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39391 (local address 127.0.97.62:43487)
0620 18:30:28.194902 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:28.194908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:28.195965 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 18:30:28.197684 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:28.197690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:28.197775 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:28.197892 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:28.199302 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.200248 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.202024 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.202631 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.202759 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:28.206038 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:28.206047 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:28.206055 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:28.206093 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:28.208652 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.209308 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.209316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.209322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.209437 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.209860 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.209865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.209868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.210110 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:28.210253 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:28.210906 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 18:30:28.211113 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:30:28.212355 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:28.195117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:39391)
0620 18:30:28.195810 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:28.195849 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:28.197432 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:28.198126 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:28.198135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:28.198613 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:28.199142 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:28.199158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.200387 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.200396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:28.201860 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:28.201874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.202853 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.202859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:28.203161 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:28.203785 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:28.203811 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:28.205817 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:28.208785 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:28.208794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:28.208809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:28.209195 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:28.209559 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:28.209563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:28.209567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:28.209748 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:28.210225 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:28.210232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:28.210550 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:30:28.210798 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 18:30:28.211271 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":246,"threads_started":1}
I20260620 18:30:28.213611 9123 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43487
I20260620 18:30:28.213950 9123 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:28.214741 9123 heartbeater.cc:511] Master 127.0.97.62:43487 requested a full tablet report, sending...
I20260620 18:30:28.216131 8667 ts_manager.cc:194] Registered new tserver with Master: 967e81844f4f418ba8974df3423fa0cd (127.0.97.3:38231)
I20260620 18:30:28.216929 8667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39391
I20260620 18:30:28.222836 8856 heartbeater.cc:503] Master 127.0.97.62:43487 was elected leader, sending a full tablet report...
I20260620 18:30:28.227350 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 18:30:28.363086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8728
Jun 20 18:30:28 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:30:28.379822 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:28.364289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.0.97.1:34137)
0620 18:30:28.364860 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:28.364868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:28.364896 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:30:28.365013 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:28.365021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:28.365304 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:28.365454 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:28.366875 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.367958 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.369312 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.369877 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.369986 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:28.374158 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:28.374169 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:28.374183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:28.374256 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:28.377129 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.377796 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.377804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.377806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.377950 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.378271 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.378274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.378276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.378603 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:28.378828 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:28.379153 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:30:28.379376 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
I20260620 18:30:28.381022 8851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:28.464116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8728
I20260620 18:30:28.482077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8862
Jun 20 18:30:28 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:30:28.497977 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:28.483026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.2:44949)
0620 18:30:28.483639 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:28.483646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:28.483672 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:30:28.483744 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:28.483752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:28.483985 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:28.484116 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:28.485343 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.486346 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.487671 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.488199 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.488304 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:28.492204 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:28.492215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:28.492229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:28.492300 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:28.495075 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.495995 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.496003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.496005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.496128 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.496489 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.496492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.496495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.496863 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:28.497093 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:28.497392 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:30:28.497581 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
I20260620 18:30:28.499267 8985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:28.582165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8862
I20260620 18:30:28.598349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8995
Jun 20 18:30:28 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:30:28.614506 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:28.599487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.97.3:38231)
0620 18:30:28.600195 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:28.600202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:28.600229 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:30:28.600287 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:28.600322 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:28.600533 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:28.600662 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:28.602030 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.603053 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.604499 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:28.605043 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:28.605139 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:28.609367 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:28.609378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:28.609392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:28.609448 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:28.612172 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.612807 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.612814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.612817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.612938 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:28.613294 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:28.613298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:28.613300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:28.613602 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:28.613842 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:28.614060 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:30:28.614240 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":370,"threads_started":1}
I20260620 18:30:28.615810 9118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:28.697353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8995
I20260620 18:30:28.713166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 8635
I20260620 18:30:28.714392 8697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:28.819790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 8635
I20260620 18:30:28.840238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43487
--webserver_interface=127.0.97.62
--webserver_port=34657
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:29.162022 9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:29.162328 9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:29.162508 9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:29.170126 9140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:29.170224 9140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:29.170321 9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:29.170421 9140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:29.170485 9140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:29.180150 9140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:29.181433 9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:29.182766 9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:29.194336 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:29.194392 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:29.194413 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:29.195449 9140 server_base.cc:1034] running on GCE node
I20260620 18:30:29.196185 9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:29.197518 9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:29.198741 9140 hybrid_clock.cc:648] HybridClock initialized: now 1781980229198690 us; error 57 us; skew 500 ppm
I20260620 18:30:29.199111 9140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:29.202164 9140 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:30:29.203466 9140 webserver.cc:460] Webserver started at http://127.0.97.62:34657/ using document root <none> and password file <none>
I20260620 18:30:29.204152 9140 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:29.204265 9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:29.209296 9140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 18:30:29.211910 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:29.213157 9140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:30:29.213307 9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "32aa04146ea544d4b2b2b21bbba29f9e"
format_stamp: "Formatted at 2026-06-20 18:30:26 on dist-test-slave-mcnm"
server_key: "677af09bd4594567d1e5d34d4e4e9143"
server_key_iv: "d9ae018cf2e5d73f6d286f42c9895e18"
server_key_version: "encryptionkey@0"
I20260620 18:30:29.214205 9140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:29.230700 9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:29.234747 9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:29.235013 9140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:29.253088 9140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43487
I20260620 18:30:29.253121 9206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43487 every 8 connection(s)
I20260620 18:30:29.254541 9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:29.261153 9207 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:30:29.261312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9140
I20260620 18:30:29.262094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34137
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43487
--webserver_port=36115
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:30:29.264479 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Bootstrap starting.
I20260620 18:30:29.289557 9207 log.cc:824] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:29.300120 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:30:29.300752 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Bootstrap complete.
I20260620 18:30:29.313249 9207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:29.314391 9207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa04146ea544d4b2b2b21bbba29f9e, State: Initialized, Role: FOLLOWER
I20260620 18:30:29.315122 9207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:29.315407 9207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:29.315539 9207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:29.315722 9207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:30:29.321384 9207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:29.321942 9207 leader_election.cc:302] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32aa04146ea544d4b2b2b21bbba29f9e; no voters:
I20260620 18:30:29.322547 9207 leader_election.cc:288] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:30:29.322685 9211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:30:29.324040 9211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [term 2 LEADER]: Becoming Leader. State: Replica: 32aa04146ea544d4b2b2b21bbba29f9e, State: Running, Role: LEADER
I20260620 18:30:29.324635 9207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:29.324581 9211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } }
I20260620 18:30:29.327370 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } } }
I20260620 18:30:29.327365 9213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32aa04146ea544d4b2b2b21bbba29f9e. Latest consensus state: current_term: 2 leader_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa04146ea544d4b2b2b21bbba29f9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43487 } } }
I20260620 18:30:29.327800 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:29.327828 9213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:29.342583 9218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:29.347968 9218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:29.349350 9218 catalog_manager.cc:1259] Loaded cluster ID: 041abe2e9a344c7c859d2fbcfe1d87a4
I20260620 18:30:29.349493 9218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:29.353499 9218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:29.354949 9218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32aa04146ea544d4b2b2b21bbba29f9e: Loaded TSK: 0
I20260620 18:30:29.356452 9218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:30:29.617277 9209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:29.617540 9209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:29.617684 9209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:29.625237 9209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:29.625339 9209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:29.625506 9209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:29.635202 9209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36115
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:29.636489 9209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:29.637806 9209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:29.651698 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:29.651823 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:29.652029 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:29.653481 9209 server_base.cc:1034] running on GCE node
I20260620 18:30:29.654053 9209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:29.654971 9209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:29.656209 9209 hybrid_clock.cc:648] HybridClock initialized: now 1781980229656130 us; error 50 us; skew 500 ppm
I20260620 18:30:29.656522 9209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:29.659724 9209 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 18:30:29.661232 9209 webserver.cc:460] Webserver started at http://127.0.97.1:36115/ using document root <none> and password file <none>
I20260620 18:30:29.661901 9209 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:29.662036 9209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:29.667400 9209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:29.670132 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:29.671502 9209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:29.671725 9209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "dfca6bdd8cbf45d6bf42a20e49098030"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "65efeb168c0c89830117945b2e6d9a56"
server_key_iv: "35db0349bf3ec2c2104abd392ecd54d9"
server_key_version: "encryptionkey@0"
I20260620 18:30:29.672613 9209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:29.688720 9209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:29.692982 9209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:29.693257 9209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:29.694224 9209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:29.695621 9209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:29.695734 9209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:29.695825 9209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:29.695881 9209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:29.729522 9209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34137
I20260620 18:30:29.729598 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34137 every 8 connection(s)
I20260620 18:30:29.730938 9209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:29.737228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9209
I20260620 18:30:29.737989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44949
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43487
--webserver_port=35105
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:29.751052 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:29.733889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:56319)
0620 18:30:29.734649 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:29.734699 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:29.736425 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:29.737889 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:29.737900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:29.738740 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:29.739561 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.739591 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.741556 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.741564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.743746 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.743764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.745198 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.745208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.745492 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.746203 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:29.746229 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:29.750108 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":326,"threads_started":1}
W20260620 18:30:29.751724 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:29.733813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56319 (local address 127.0.97.62:43487)
0620 18:30:29.734455 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:29.734466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:29.734810 (+ 344us) server_negotiation.cc:407] Connection header received
0620 18:30:29.736972 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:29.736987 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:29.737369 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:29.737682 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:29.739765 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.741371 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.743887 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.744931 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.745087 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.750545 (+ 5458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56319: BlockingRecv error: recv got EOF from 127.0.97.1:56319 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":253,"threads_started":1}
W20260620 18:30:29.751770 9356 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43487: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:30:29.760181 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:29.752545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55343 (local address 127.0.97.62:43487)
0620 18:30:29.752728 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:29.752735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:29.752752 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:29.752802 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:29.752806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:29.752857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:29.752971 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:29.753568 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.754397 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.755570 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.755961 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.756091 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.760016 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55343: BlockingRecv error: recv got EOF from 127.0.97.1:55343 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:30:29.760180 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:29.752328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:55343)
0620 18:30:29.752536 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:29.752549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:29.752657 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:29.752990 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:29.752995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:29.753321 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:29.753445 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.753452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.754527 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.754533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.755414 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.755425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.756096 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.756102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.756281 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.756875 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:29.756891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:29.759836 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:29 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:29.768549 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:29.761052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:39189)
0620 18:30:29.761228 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:29.761248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:29.761353 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:29.761649 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:29.761654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:29.761950 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:29.762087 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.762095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.763017 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.763022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.763983 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:29.763993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.764708 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.764714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:29.764877 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.765468 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:29.765485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:29.768262 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:30:29.768661 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:29.761185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39189 (local address 127.0.97.62:43487)
0620 18:30:29.761346 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:29.761350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:29.761367 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:30:29.761454 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:29.761458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:29.761532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:29.761628 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:29.762214 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.762886 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.764114 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:29.764564 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:29.764690 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:29.768435 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39189: BlockingRecv error: recv got EOF from 127.0.97.1:39189 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 18:30:29.768849 9356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:30:30.061743 9360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:30.062037 9360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:30.062180 9360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:30.069365 9360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:30.069530 9360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:30.069689 9360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:30.079824 9360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35105
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:30.081125 9360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:30.082335 9360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:30.095165 9365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:30.095082 9368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:30.095126 9366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:30.095942 9360 server_base.cc:1034] running on GCE node
I20260620 18:30:30.096516 9360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:30.097398 9360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:30.098629 9360 hybrid_clock.cc:648] HybridClock initialized: now 1781980230098581 us; error 59 us; skew 500 ppm
I20260620 18:30:30.098942 9360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:30.102026 9360 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 18:30:30.103405 9360 webserver.cc:460] Webserver started at http://127.0.97.2:35105/ using document root <none> and password file <none>
I20260620 18:30:30.104105 9360 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:30.104264 9360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:30.109397 9360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:30:30.112217 9375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.113466 9360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:30:30.113627 9360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "acacd94d66b44f6283dc7c103b2ef878"
format_stamp: "Formatted at 2026-06-20 18:30:27 on dist-test-slave-mcnm"
server_key: "da1b3cee32e6ac25427e09a85f163227"
server_key_iv: "70c88a9002d3de71a773cc4983187a3c"
server_key_version: "encryptionkey@0"
I20260620 18:30:30.114535 9360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:30.145725 9360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:30.150168 9360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:30.150444 9360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:30.151422 9360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:30.152721 9360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:30.152846 9360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.152969 9360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:30.153051 9360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.185714 9360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44949
I20260620 18:30:30.185801 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44949 every 8 connection(s)
I20260620 18:30:30.187109 9360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:30.191762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9360
I20260620 18:30:30.192380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38231
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43487
--webserver_port=32953
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:30:30.202694 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.189689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39735 (local address 127.0.97.62:43487)
0620 18:30:30.189902 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.189910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.191158 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 18:30:30.193141 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.193147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.193232 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.193347 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.194794 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.195482 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.197024 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.197452 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.197586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.202451 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39735: BlockingRecv error: recv got EOF from 127.0.97.2:39735 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:30:30.203389 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.190209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:39735)
0620 18:30:30.190997 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.191040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.192887 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.193577 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.193586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.194074 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.194613 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.194628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.195668 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.195672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.196876 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.196888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.197672 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.197680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.197908 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.198609 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.198630 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.202277 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":330,"threads_started":1}
W20260620 18:30:30.204051 9488 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43487: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:30.212206 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.204674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:37479)
0620 18:30:30.204848 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.204868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.204968 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.205278 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.205282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.205616 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.205752 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.205767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.206711 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.206717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.207657 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.207668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.208322 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.208327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.208497 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.209084 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.209100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.211929 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:30:30.212499 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.204789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37479 (local address 127.0.97.62:43487)
0620 18:30:30.204952 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.204957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.204975 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:30.205094 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.205098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.205162 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.205257 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.205887 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.206573 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.207787 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.208179 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.208312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.212133 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37479: BlockingRecv error: recv got EOF from 127.0.97.2:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:30.220427 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.212983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:59877)
0620 18:30:30.213135 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.213160 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.213255 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.213532 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.213536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.213822 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.213953 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.213960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.214891 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.214897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.215854 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.215865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.216501 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.216506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.216722 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.217275 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.217290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.220139 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 18:30:30.220551 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.213093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59877 (local address 127.0.97.62:43487)
0620 18:30:30.213278 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.213286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.213305 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:30:30.213353 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.213358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.213421 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.213515 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.214080 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.214767 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.215982 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.216368 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.216492 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.220328 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59877: BlockingRecv error: recv got EOF from 127.0.97.2:59877 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 18:30:30.220898 9488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:30:30.511168 9491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:30.511480 9491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:30.511665 9491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:30.518959 9491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:30.519097 9491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:30.519276 9491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:30.528769 9491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32953
--tserver_master_addrs=127.0.97.62:43487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:30.530040 9491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:30.531320 9491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:30.544342 9499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:30.544384 9496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:30.544306 9497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:30.545195 9491 server_base.cc:1034] running on GCE node
I20260620 18:30:30.545840 9491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:30.546824 9491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:30.548046 9491 hybrid_clock.cc:648] HybridClock initialized: now 1781980230547987 us; error 52 us; skew 500 ppm
I20260620 18:30:30.548365 9491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:30.551333 9491 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 18:30:30.552762 9491 webserver.cc:460] Webserver started at http://127.0.97.3:32953/ using document root <none> and password file <none>
I20260620 18:30:30.553432 9491 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:30.553582 9491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:30.558950 9491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 18:30:30.561827 9506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.563220 9491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:30.563406 9491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "967e81844f4f418ba8974df3423fa0cd"
format_stamp: "Formatted at 2026-06-20 18:30:28 on dist-test-slave-mcnm"
server_key: "eaa126d8b131292549f04ce4a2649702"
server_key_iv: "3a6cbd572fc9ed89fabd2b4017cb4794"
server_key_version: "encryptionkey@0"
I20260620 18:30:30.564425 9491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:30.593590 9491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:30.597805 9491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:30.598083 9491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:30.599061 9491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:30.600473 9491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:30.600605 9491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.600731 9491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:30.600823 9491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:30.634105 9491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38231
I20260620 18:30:30.634188 9618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38231 every 8 connection(s)
I20260620 18:30:30.635543 9491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:30.644183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9491
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:30:30.653103 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.638268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55619 (local address 127.0.97.62:43487)
0620 18:30:30.638471 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.638476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.639756 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 18:30:30.641550 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.641555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.641650 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.641755 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.643215 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.643996 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.646180 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.646831 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.646969 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.652853 (+ 5884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55619: BlockingRecv error: recv got EOF from 127.0.97.3:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:30:30.654019 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.638806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:55619)
0620 18:30:30.639538 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.639628 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.641285 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.641986 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.641994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.642501 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.643041 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.643056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.644140 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.644150 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.645986 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.646002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.647056 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.647066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.647420 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.648314 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.648338 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.652692 (+ 4354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
W20260620 18:30:30.654697 9619 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43487: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260620 18:30:30.661662 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.645274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.97.62:43487)
0620 18:30:30.645959 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.645965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.645983 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:30:30.646035 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.646043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.646119 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.646241 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.648062 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.648840 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.650487 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.651226 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.651332 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.654480 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:30.654492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:30.654511 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:30.654556 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:30.658488 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:30.659131 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:30.659138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:30.659144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:30.659343 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:30.659731 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:30.659736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:30.659739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:30.660163 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:30.660467 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:30.660907 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 18:30:30.661268 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":447,"threads_started":1}
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:30.664021 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.655561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:52425)
0620 18:30:30.655720 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.655739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.655828 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.656181 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.656184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.656549 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.656730 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.656739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.658161 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.658165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.659014 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.659024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.659783 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.659788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.659961 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.660467 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.660483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.663739 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:30:30.664636 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.655334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52425 (local address 127.0.97.62:43487)
0620 18:30:30.655549 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.655554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.655813 (+ 259us) server_negotiation.cc:407] Connection header received
0620 18:30:30.655919 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.655924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.656050 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.656164 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.656871 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.658010 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.659165 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.659646 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.659777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.663914 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52425: BlockingRecv error: recv got EOF from 127.0.97.3:52425 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:30.672194 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.664792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:49163)
0620 18:30:30.664970 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.664991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.665085 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.665550 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.665554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.665816 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.665954 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.665960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.666899 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.666902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.667924 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.667935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.668626 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.668630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.668754 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.669263 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.669278 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.671801 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:30:30.672260 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.665068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49163 (local address 127.0.97.62:43487)
0620 18:30:30.665279 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.665283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.665297 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:30:30.665367 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.665373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.665440 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.665532 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.666099 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.666784 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.668075 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.668455 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.668613 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.671987 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49163: BlockingRecv error: recv got EOF from 127.0.97.3:49163 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260620 18:30:30.672598 9619 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:30 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:30.778236 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:30.769476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:38191)
0620 18:30:30.769988 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:30.770004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:30.770156 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:30.770489 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:30.770494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:30.770975 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:30.771170 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.771178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.772364 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.772370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.773473 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:30.773485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.774199 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.774205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:30.774360 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.774955 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:30.774972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:30.777947 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
W20260620 18:30:30.778299 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:30.769527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38191 (local address 127.0.97.62:43487)
0620 18:30:30.769771 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:30.769779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:30.770085 (+ 306us) server_negotiation.cc:407] Connection header received
0620 18:30:30.770292 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:30.770298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:30.770367 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:30.770462 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:30.771322 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.772227 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.773604 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:30.774021 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:30.774153 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:30.778128 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38191: BlockingRecv error: recv got EOF from 127.0.97.1:38191 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:31.229861 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:31.221516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:53617)
0620 18:30:31.222030 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:31.222046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:31.222195 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:31.222516 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:31.222520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:31.222992 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:31.223169 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.223177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.224192 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.224198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.225237 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.225248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.226195 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.226203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.226384 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.226959 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:31.226975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:31.229653 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
W20260620 18:30:31.230126 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:31.221590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53617 (local address 127.0.97.62:43487)
0620 18:30:31.221862 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:31.221877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:31.222128 (+ 251us) server_negotiation.cc:407] Connection header received
0620 18:30:31.222320 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:31.222329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:31.222389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:31.222489 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:31.223306 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.224062 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.225395 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.225992 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.226125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.229851 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53617: BlockingRecv error: recv got EOF from 127.0.97.2:53617 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:31.682135 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:31.673225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:55953)
0620 18:30:31.673777 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:31.673798 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:31.673953 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:31.674279 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:31.674284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:31.674809 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:31.674998 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.675007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.676070 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.676076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.677072 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.677082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.677753 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.677759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.677942 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.678568 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:31.678595 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:31.681904 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
W20260620 18:30:31.682421 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:31.673456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55953 (local address 127.0.97.62:43487)
0620 18:30:31.673680 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:31.673688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:31.673880 (+ 192us) server_negotiation.cc:407] Connection header received
0620 18:30:31.674087 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:31.674093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:31.674157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:31.674257 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:31.675140 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.675918 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.677202 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.677586 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.677719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.682146 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55953: BlockingRecv error: recv got EOF from 127.0.97.3:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:31 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:31.787352 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:31.779191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:58257)
0620 18:30:31.779776 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:31.779810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:31.779945 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:31.780284 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:31.780289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:31.780708 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:31.780847 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.780856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.781885 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.781891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.782783 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:31.782794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.783481 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.783488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:31.783674 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.784284 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:31.784302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:31.787023 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":351,"threads_started":1}
W20260620 18:30:31.787606 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:31.779240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58257 (local address 127.0.97.62:43487)
0620 18:30:31.779447 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:31.779452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:31.779879 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:30:31.780072 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:31.780077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:31.780154 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:31.780252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:31.780991 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.781746 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.782920 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:31.783311 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:31.783447 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:31.787202 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58257: BlockingRecv error: recv got EOF from 127.0.97.1:58257 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:32.243777 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:32.230808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:33751)
0620 18:30:32.231421 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:32.231437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:32.231617 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:32.231974 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:32.231978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:32.232403 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:32.232542 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.232551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.233554 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.233560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.234556 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.234568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.235263 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.235269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.235430 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.240441 (+ 5011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:32.240470 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:32.243488 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":386,"threads_started":1}
W20260620 18:30:32.244038 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:32.230886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33751 (local address 127.0.97.62:43487)
0620 18:30:32.231234 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:32.231265 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:32.231548 (+ 283us) server_negotiation.cc:407] Connection header received
0620 18:30:32.231748 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:32.231753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:32.231819 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:32.231925 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:32.232690 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.233417 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.234692 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.235119 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.235246 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.243758 (+ 8512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33751: BlockingRecv error: recv got EOF from 127.0.97.2:33751 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:32.692776 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:32.683228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:52427)
0620 18:30:32.683855 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:32.683874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:32.683999 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:32.684414 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:32.684419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:32.684867 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:32.685038 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.685047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.686228 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.686234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.687272 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.687316 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.688388 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.688397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.688609 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.689283 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:32.689302 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:32.692532 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":365,"threads_started":1}
W20260620 18:30:32.693013 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:32.683469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52427 (local address 127.0.97.62:43487)
0620 18:30:32.683673 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:32.683677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:32.683982 (+ 305us) server_negotiation.cc:407] Connection header received
0620 18:30:32.684155 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:32.684160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:32.684226 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:32.684321 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:32.685227 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.685971 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.687465 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.688078 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.688206 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.692756 (+ 4550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52427: BlockingRecv error: recv got EOF from 127.0.97.3:52427 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:32 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:32.796653 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:32.788195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:45009)
0620 18:30:32.788736 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:32.788752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:32.788883 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:32.789187 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:32.789190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:32.789520 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:32.789685 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.789693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.790743 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.790748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.791734 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:32.791747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.792460 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.792465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:32.792610 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.793294 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:32.793312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:32.796384 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":303,"threads_started":1}
W20260620 18:30:32.796926 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:32.788274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45009 (local address 127.0.97.62:43487)
0620 18:30:32.788434 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:32.788438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:32.788821 (+ 383us) server_negotiation.cc:407] Connection header received
0620 18:30:32.789005 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:32.789009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:32.789068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:32.789143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:32.789837 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.790597 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.791874 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:32.792290 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:32.792396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:32.796595 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45009: BlockingRecv error: recv got EOF from 127.0.97.1:45009 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:33.253679 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:33.244784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:41137)
0620 18:30:33.245320 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:33.245335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:33.245517 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:33.245838 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:33.245842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:33.246213 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:33.246394 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.246404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.247586 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.247591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.248922 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.248939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.249671 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.249675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.249811 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.250464 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:33.250482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:33.253358 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
W20260620 18:30:33.253707 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:33.244923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41137 (local address 127.0.97.62:43487)
0620 18:30:33.245158 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:33.245166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:33.245438 (+ 272us) server_negotiation.cc:407] Connection header received
0620 18:30:33.245652 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:33.245656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:33.245713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:33.245809 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:33.246541 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.247404 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.249054 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.249536 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.249636 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.253516 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41137: BlockingRecv error: recv got EOF from 127.0.97.2:41137 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:33.702287 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:33.693772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:44997)
0620 18:30:33.694369 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:33.694386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:33.694510 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:33.694866 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:33.694871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:33.695293 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:33.695516 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.695529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.696686 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.696691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.697705 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.697717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.698383 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.698387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.698533 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.699162 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:33.699178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:33.701921 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":303,"threads_started":1}
W20260620 18:30:33.702364 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:33.693918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44997 (local address 127.0.97.62:43487)
0620 18:30:33.694082 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:33.694086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:33.694470 (+ 384us) server_negotiation.cc:407] Connection header received
0620 18:30:33.694611 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:33.694615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:33.694678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:33.694764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:33.695718 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.696576 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.697831 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.698233 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.698333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.702149 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44997: BlockingRecv error: recv got EOF from 127.0.97.3:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:33 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:33.805982 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:33.797854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:46527)
0620 18:30:33.798339 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:33.798356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:33.798470 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:33.798788 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:33.798791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:33.799129 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:33.799271 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.799279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.800345 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.800349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.801334 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:33.801344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.802047 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.802051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:33.802196 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.802790 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:33.802810 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:33.805625 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":253,"threads_started":1}
W20260620 18:30:33.806089 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:33.797886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46527 (local address 127.0.97.62:43487)
0620 18:30:33.798120 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:33.798131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:33.798445 (+ 314us) server_negotiation.cc:407] Connection header received
0620 18:30:33.798572 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:33.798576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:33.798641 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:33.798738 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:33.799420 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.800206 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.801467 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:33.801900 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:33.802001 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:33.805850 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46527: BlockingRecv error: recv got EOF from 127.0.97.1:46527 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:34.264130 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:34.254670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:33577)
0620 18:30:34.255363 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:34.255384 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:34.255542 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:34.255956 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:34.255961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:34.256399 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:34.256582 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.256593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.257637 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.257643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.259164 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.259180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.260018 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.260024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.260206 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.260862 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:34.260886 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:34.263916 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":342,"threads_started":1}
W20260620 18:30:34.264283 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:34.254766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33577 (local address 127.0.97.62:43487)
0620 18:30:34.255032 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:34.255041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:34.255656 (+ 615us) server_negotiation.cc:407] Connection header received
0620 18:30:34.255720 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:34.255724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:34.255820 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:34.255905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:34.256741 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.257504 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.259311 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.259877 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.259995 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.264071 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33577: BlockingRecv error: recv got EOF from 127.0.97.2:33577 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:34.711787 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:34.703220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:35247)
0620 18:30:34.703725 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:34.703742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:34.703855 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:34.704138 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:34.704143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:34.704498 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:34.704658 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.704666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.705705 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.705710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.706659 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.706670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.707398 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.707402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.707585 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.708237 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:34.708256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:34.711395 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":278,"threads_started":1}
W20260620 18:30:34.711941 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:34.703261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35247 (local address 127.0.97.62:43487)
0620 18:30:34.703464 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:34.703474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:34.703814 (+ 340us) server_negotiation.cc:407] Connection header received
0620 18:30:34.703949 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:34.703952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:34.704019 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:34.704098 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:34.704799 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.705577 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.706806 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.707269 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.707393 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.711647 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35247: BlockingRecv error: recv got EOF from 127.0.97.3:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:34 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:34.815642 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:34.807154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:60361)
0620 18:30:34.807878 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:34.807895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:34.808048 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:34.808388 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:34.808392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:34.808930 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:34.809140 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.809150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.810201 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.810206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.811131 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:34.811140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.811862 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.811867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:34.812034 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.812618 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:34.812635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:34.815276 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":296,"threads_started":1}
W20260620 18:30:34.815728 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:34.807337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60361 (local address 127.0.97.62:43487)
0620 18:30:34.807637 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:34.807646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:34.807983 (+ 337us) server_negotiation.cc:407] Connection header received
0620 18:30:34.808178 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:34.808184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:34.808248 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:34.808359 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:34.809293 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.810067 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.811268 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:34.811724 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:34.811853 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:34.815489 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60361: BlockingRecv error: recv got EOF from 127.0.97.1:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:35.273535 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:35.265104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:51779)
0620 18:30:35.265609 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:35.265625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:35.265783 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:35.266096 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:35.266100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:35.266506 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:35.266667 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.266677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.267801 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.267808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.268841 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.268853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.269514 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.269520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.269684 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.270333 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:35.270350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:35.273202 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":239,"threads_started":1}
W20260620 18:30:35.273639 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:35.265308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51779 (local address 127.0.97.62:43487)
0620 18:30:35.265516 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:35.265524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:35.265720 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:30:35.265921 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:35.265925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:35.265980 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:35.266073 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:35.266798 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.267670 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.268970 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.269378 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.269504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.273382 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51779: BlockingRecv error: recv got EOF from 127.0.97.2:51779 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:35.722281 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:35.712808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:47077)
0620 18:30:35.713357 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:35.713375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:35.713501 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:35.713808 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:35.713811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:35.714208 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:35.714386 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.714395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.715562 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.715572 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.716897 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.716913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.717895 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.717900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.718042 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.718845 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:35.718864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:35.722026 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":274,"threads_started":1}
W20260620 18:30:35.722460 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:35.712958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47077 (local address 127.0.97.62:43487)
0620 18:30:35.713183 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:35.713189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:35.713451 (+ 262us) server_negotiation.cc:407] Connection header received
0620 18:30:35.713613 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:35.713617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:35.713683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:35.713764 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:35.714568 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.715347 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.717064 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.717705 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.717840 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.722235 (+ 4395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47077: BlockingRecv error: recv got EOF from 127.0.97.3:47077 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:35 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:35.825445 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:35.816679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:57819)
0620 18:30:35.817204 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:35.817219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:35.817377 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:35.817769 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:35.817776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:35.818211 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:35.818355 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.818363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.819426 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.819434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.820502 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:35.820513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.821219 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.821225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:35.821401 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.822023 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:35.822039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:35.825143 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
W20260620 18:30:35.825582 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:35.816768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57819 (local address 127.0.97.62:43487)
0620 18:30:35.816997 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:35.817005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:35.817303 (+ 298us) server_negotiation.cc:407] Connection header received
0620 18:30:35.817501 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:35.817506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:35.817568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:35.817729 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:35.818491 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.819297 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.820650 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:35.821055 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:35.821227 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:35.825358 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57819: BlockingRecv error: recv got EOF from 127.0.97.1:57819 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:36.283022 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:36.274441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:40855)
0620 18:30:36.274983 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:36.275000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:36.275197 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:36.275555 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:36.275560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:36.275976 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:36.276182 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.276192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.277278 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.277287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.278182 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.278192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.279007 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.279012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.279147 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.279768 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:36.279786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:36.282709 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":301,"threads_started":1}
W20260620 18:30:36.283252 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:36.274569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40855 (local address 127.0.97.62:43487)
0620 18:30:36.274794 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:36.274802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:36.275110 (+ 308us) server_negotiation.cc:407] Connection header received
0620 18:30:36.275320 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:36.275325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:36.275394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:36.275494 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:36.276327 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.277130 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.278354 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.278747 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.278859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.282976 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40855: BlockingRecv error: recv got EOF from 127.0.97.2:40855 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:36.731853 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:36.723294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:36381)
0620 18:30:36.723872 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:36.723893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:36.724031 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:36.724387 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:36.724392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:36.724918 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:36.725059 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.725068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.726170 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.726173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.727070 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.727080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.727885 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.727889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.728003 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.728652 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:36.728669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:36.731649 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
W20260620 18:30:36.732136 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:36.723367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36381 (local address 127.0.97.62:43487)
0620 18:30:36.723595 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:36.723603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:36.723997 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:30:36.724151 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:36.724156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:36.724241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:36.724332 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:36.725201 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.726006 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.727209 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.727736 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.727855 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.731896 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36381: BlockingRecv error: recv got EOF from 127.0.97.3:36381 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:36 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:36.834986 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:36.826303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:54985)
0620 18:30:36.826933 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:36.826950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:36.827115 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:36.827447 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:36.827452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:36.827895 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:36.828062 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.828070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.829066 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.829073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.829995 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:36.830006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.830759 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.830765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:36.830937 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.831829 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:36.831853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:36.834738 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":276,"threads_started":1}
W20260620 18:30:36.835130 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:36.826385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54985 (local address 127.0.97.62:43487)
0620 18:30:36.826615 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:36.826623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:36.827042 (+ 419us) server_negotiation.cc:407] Connection header received
0620 18:30:36.827242 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:36.827247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:36.827314 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:36.827418 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:36.828209 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.828920 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.830168 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:36.830603 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:36.830745 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:36.834924 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54985: BlockingRecv error: recv got EOF from 127.0.97.1:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:37.296181 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:37.284069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:42549)
0620 18:30:37.284717 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:37.284735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:37.284856 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:37.285222 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:37.285225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:37.285655 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:37.285819 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.285827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.286865 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.286869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.287925 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.287938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.288626 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.288631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.288745 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.292907 (+ 4162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:37.292925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:37.295946 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
W20260620 18:30:37.296342 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:37.284154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42549 (local address 127.0.97.62:43487)
0620 18:30:37.284347 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:37.284351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:37.284816 (+ 465us) server_negotiation.cc:407] Connection header received
0620 18:30:37.285001 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:37.285005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:37.285086 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:37.285172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:37.285954 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.286719 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.288055 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.288421 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.288511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.296118 (+ 7607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42549: BlockingRecv error: recv got EOF from 127.0.97.2:42549 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:37.741711 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:37.733018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:34841)
0620 18:30:37.733647 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:37.733664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:37.733773 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:37.734139 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:37.734143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:37.734565 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:37.734735 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.734743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.735920 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.735925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.736860 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.736872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.737571 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.737575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.737708 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.738376 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:37.738393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:37.741401 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":258,"threads_started":1}
W20260620 18:30:37.741966 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:37.733110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34841 (local address 127.0.97.62:43487)
0620 18:30:37.733264 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:37.733268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:37.733737 (+ 469us) server_negotiation.cc:407] Connection header received
0620 18:30:37.733938 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:37.733942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:37.734013 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:37.734094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:37.734860 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.735799 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.737008 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.737419 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.737524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.741719 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34841: BlockingRecv error: recv got EOF from 127.0.97.3:34841 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:37 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:37.845037 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:37.836048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:54957)
0620 18:30:37.836670 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:37.836695 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:37.836870 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:37.837236 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:37.837240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:37.837677 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:37.837826 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.837834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.839001 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.839009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.840137 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:37.840148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.840891 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.840897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:37.841069 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.841681 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:37.841703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:37.844701 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":307,"threads_started":1}
W20260620 18:30:37.845175 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:37.836344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54957 (local address 127.0.97.62:43487)
0620 18:30:37.836619 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:37.836625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:37.836790 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:30:37.837022 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:37.837028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:37.837095 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:37.837204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:37.837980 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.838863 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.840280 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:37.840710 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:37.840854 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:37.844923 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54957: BlockingRecv error: recv got EOF from 127.0.97.1:54957 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:38.305537 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:38.297051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:46599)
0620 18:30:38.297566 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:38.297583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:38.297696 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:38.297990 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:38.297993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:38.298363 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:38.298544 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.298553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.299709 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.299712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.300636 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.300647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.301318 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.301323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.301466 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.302064 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:38.302081 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:38.305169 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":271,"threads_started":1}
W20260620 18:30:38.305590 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:38.297195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46599 (local address 127.0.97.62:43487)
0620 18:30:38.297365 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:38.297370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:38.297657 (+ 287us) server_negotiation.cc:407] Connection header received
0620 18:30:38.297792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:38.297796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:38.297856 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:38.297946 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:38.298675 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.299548 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.300751 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.301166 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.301256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.305360 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46599: BlockingRecv error: recv got EOF from 127.0.97.2:46599 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:38.752911 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:38.742865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:45591)
0620 18:30:38.743492 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:38.743530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:38.743685 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:38.744035 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:38.744039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:38.744465 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:38.744614 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.744624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.745829 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.745839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.747544 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.747558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.748627 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.748632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.748758 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.749401 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:38.749421 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:38.752668 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":385,"threads_started":1}
W20260620 18:30:38.753175 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:38.743035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45591 (local address 127.0.97.62:43487)
0620 18:30:38.743252 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:38.743259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:38.743615 (+ 356us) server_negotiation.cc:407] Connection header received
0620 18:30:38.743825 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:38.743832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:38.743901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:38.744004 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:38.744765 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.745530 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.747746 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.748429 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.748580 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.752887 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45591: BlockingRecv error: recv got EOF from 127.0.97.3:45591 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:38 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:38.855237 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:38.846177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:50949)
0620 18:30:38.846748 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:38.846764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:38.846929 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:38.847357 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:38.847361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:38.847794 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:38.847975 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.847983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.849013 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.849020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.849994 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:38.850005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.850780 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.850786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:38.850964 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.851634 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:38.851651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:38.854863 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":311,"threads_started":1}
W20260620 18:30:38.855338 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:38.846246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50949 (local address 127.0.97.62:43487)
0620 18:30:38.846457 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:38.846466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:38.846860 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:30:38.847080 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:38.847084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:38.847194 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:38.847308 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:38.848120 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.848868 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.850146 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:38.850618 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:38.850769 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:38.855111 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50949: BlockingRecv error: recv got EOF from 127.0.97.1:50949 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:39.315233 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:39.306617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:49931)
0620 18:30:39.307221 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:39.307246 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:39.307400 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:39.307811 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:39.307816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:39.308341 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:39.308507 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.308516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.309584 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.309590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.310528 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.310538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.311263 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.311269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.311444 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.312079 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:39.312096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:39.314981 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":324,"threads_started":1}
W20260620 18:30:39.315448 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:39.306739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49931 (local address 127.0.97.62:43487)
0620 18:30:39.306941 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:39.306948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:39.307374 (+ 426us) server_negotiation.cc:407] Connection header received
0620 18:30:39.307571 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:39.307577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:39.307661 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:39.307765 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:39.308684 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.309444 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.310672 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.311103 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.311236 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.315196 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49931: BlockingRecv error: recv got EOF from 127.0.97.2:49931 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:39.765529 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:39.753913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:51853)
0620 18:30:39.754403 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:39.754420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:39.754542 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:39.754906 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:39.754910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:39.755344 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:39.755571 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.755581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.756714 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.756718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.757832 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.757844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.758567 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.758571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.758699 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.759403 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:39.759423 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:39.764790 (+ 5367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
W20260620 18:30:39.765542 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:39.754066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51853 (local address 127.0.97.62:43487)
0620 18:30:39.754288 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:39.754295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:39.754499 (+ 204us) server_negotiation.cc:407] Connection header received
0620 18:30:39.754637 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:39.754641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:39.754702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:39.754811 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:39.755834 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.756580 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.757996 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.758425 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.758524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.765229 (+ 6705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51853: BlockingRecv error: recv got EOF from 127.0.97.3:51853 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:39 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:39.865049 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:39.856376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:35769)
0620 18:30:39.856947 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:39.856964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:39.857123 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:39.857483 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:39.857488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:39.857886 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:39.858022 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.858030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.859038 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.859046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.860181 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:39.860195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.861031 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.861037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:39.861212 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.861825 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:39.861840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:39.864834 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":295,"threads_started":1}
W20260620 18:30:39.865314 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:39.856595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35769 (local address 127.0.97.62:43487)
0620 18:30:39.856863 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:39.856869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:39.857053 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:30:39.857250 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:39.857256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:39.857345 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:39.857452 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:39.858172 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.858909 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.860323 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:39.860882 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:39.861031 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:39.865031 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35769: BlockingRecv error: recv got EOF from 127.0.97.1:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:40.325106 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:40.316241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:35695)
0620 18:30:40.316732 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:40.316748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:40.316854 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:40.317185 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:40.317188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:40.317656 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:40.317844 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.317854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.318948 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.318951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.320505 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.320517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.321318 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.321322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.321436 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.322005 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:40.322022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:40.324757 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":286,"threads_started":1}
W20260620 18:30:40.325217 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:40.316309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35695 (local address 127.0.97.62:43487)
0620 18:30:40.316500 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:40.316504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:40.316820 (+ 316us) server_negotiation.cc:407] Connection header received
0620 18:30:40.316992 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:40.316996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:40.317055 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:40.317144 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:40.317992 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.318821 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.320625 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.321189 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.321282 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.324959 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35695: BlockingRecv error: recv got EOF from 127.0.97.2:35695 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:40.777565 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:40.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:49543)
0620 18:30:40.767570 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:40.767595 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:40.767767 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:40.768122 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:40.768128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:40.768709 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:40.768928 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.768943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.770685 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.770694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.771878 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.771889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.772823 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.772830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.773018 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.773710 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:40.773737 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:40.777307 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":358,"threads_started":1}
W20260620 18:30:40.777736 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:40.767256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49543 (local address 127.0.97.62:43487)
0620 18:30:40.767458 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:40.767466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:40.767690 (+ 224us) server_negotiation.cc:407] Connection header received
0620 18:30:40.767894 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:40.767898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:40.767959 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:40.768039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:40.769160 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.770453 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.772005 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.772417 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.772563 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.777508 (+ 4945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49543: BlockingRecv error: recv got EOF from 127.0.97.3:49543 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:40 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:40.875039 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:40.866092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:55655)
0620 18:30:40.866690 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:40.866709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:40.866863 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:40.867238 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:40.867243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:40.867829 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:40.868106 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.868120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.869355 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.869363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.870271 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:40.870282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.871015 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.871021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:40.871197 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.871890 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:40.871907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:40.874685 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":343,"threads_started":1}
W20260620 18:30:40.875204 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:40.866220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55655 (local address 127.0.97.62:43487)
0620 18:30:40.866433 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:40.866442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:40.866788 (+ 346us) server_negotiation.cc:407] Connection header received
0620 18:30:40.867001 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:40.867006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:40.867092 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:40.867206 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:40.868250 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.869212 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.870411 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:40.870861 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:40.871000 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:40.874886 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55655: BlockingRecv error: recv got EOF from 127.0.97.1:55655 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:41.334492 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:41.326084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:57125)
0620 18:30:41.326571 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:41.326588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:41.326704 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:41.327000 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:41.327004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:41.327405 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:41.327645 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.327655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.328715 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.328719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.329834 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.329846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.330783 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.330787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.330899 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.331458 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:41.331474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:41.334266 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
W20260620 18:30:41.334758 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:41.326162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57125 (local address 127.0.97.62:43487)
0620 18:30:41.326351 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:41.326356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:41.326663 (+ 307us) server_negotiation.cc:407] Connection header received
0620 18:30:41.326806 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:41.326810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:41.326876 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:41.326969 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:41.327774 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.328525 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.329993 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.330653 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.330748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.334462 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57125: BlockingRecv error: recv got EOF from 127.0.97.2:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:41.786980 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:41.778558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:32965)
0620 18:30:41.779105 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:41.779121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:41.779237 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:41.779572 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:41.779576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:41.779974 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:41.780150 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.780158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.781190 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.781194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.782081 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.782091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.782774 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.782779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.782907 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.783647 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:41.783667 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:41.786728 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":291,"threads_started":1}
W20260620 18:30:41.787150 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:41.778620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32965 (local address 127.0.97.62:43487)
0620 18:30:41.778790 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:41.778795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:41.779222 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:30:41.779321 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:41.779324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:41.779385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:41.779467 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:41.780288 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.781064 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.782198 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.782588 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.782692 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.786908 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:32965: BlockingRecv error: recv got EOF from 127.0.97.3:32965 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:41 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:41.888929 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:41.876242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:33623)
0620 18:30:41.876855 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:41.876871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:41.877079 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:41.877499 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:41.877504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:41.878098 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:41.878385 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.878448 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.880294 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.880301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.881615 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:41.881627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.882480 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.882488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:41.882687 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.883681 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:41.883715 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:41.888406 (+ 4691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":365,"threads_started":1}
W20260620 18:30:41.889153 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:41.876270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33623 (local address 127.0.97.62:43487)
0620 18:30:41.876441 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:41.876446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:41.877036 (+ 590us) server_negotiation.cc:407] Connection header received
0620 18:30:41.877205 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:41.877208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:41.877293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:41.877464 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:41.878693 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.880144 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.881761 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:41.882305 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:41.882448 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:41.888882 (+ 6434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33623: BlockingRecv error: recv got EOF from 127.0.97.1:33623 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:42.351478 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:42.335456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:33085)
0620 18:30:42.336027 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:42.336045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:42.336164 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:42.336478 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:42.336494 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:42.337030 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:42.337281 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.337293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.338653 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.338658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.340068 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.340081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.340889 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.340894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.341024 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.347798 (+ 6774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:42.347819 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:42.351176 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":324,"threads_started":1}
W20260620 18:30:42.351702 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:42.335669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33085 (local address 127.0.97.62:43487)
0620 18:30:42.335849 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:42.335854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:42.336154 (+ 300us) server_negotiation.cc:407] Connection header received
0620 18:30:42.336273 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:42.336277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:42.336339 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:42.336435 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:42.337459 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.338470 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.340225 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.340724 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.340841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.351453 (+ 10612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33085: BlockingRecv error: recv got EOF from 127.0.97.2:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:42.796728 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:42.787998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:43523)
0620 18:30:42.788696 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:42.788714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:42.788843 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:42.789223 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:42.789227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:42.789613 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:42.789765 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.789773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.790826 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.790831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.791941 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.791954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.792695 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.792700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.792840 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.793589 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:42.793607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:42.796384 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":447,"threads_started":1}
W20260620 18:30:42.796840 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:42.788130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43523 (local address 127.0.97.62:43487)
0620 18:30:42.788303 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:42.788307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:42.788826 (+ 519us) server_negotiation.cc:407] Connection header received
0620 18:30:42.788977 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:42.788981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:42.789050 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:42.789142 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:42.789931 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.790668 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.792084 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.792512 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.792607 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.796600 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43523: BlockingRecv error: recv got EOF from 127.0.97.3:43523 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:42 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:42.899506 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:42.890866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:56195)
0620 18:30:42.891575 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:42.891602 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:42.891762 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:42.892133 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:42.892138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:42.892657 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:42.892899 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.892909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.893930 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.893936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.894857 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:42.894869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.895664 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.895671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:42.895842 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.896478 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:42.896495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:42.899178 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":356,"threads_started":1}
W20260620 18:30:42.899650 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:42.891024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56195 (local address 127.0.97.62:43487)
0620 18:30:42.891305 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:42.891313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:42.891693 (+ 380us) server_negotiation.cc:407] Connection header received
0620 18:30:42.891877 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:42.891884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:42.891969 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:42.892108 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:42.893042 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.893788 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.895005 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:42.895435 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:42.895591 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:42.899373 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56195: BlockingRecv error: recv got EOF from 127.0.97.1:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:43.361975 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:43.352563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:52003)
0620 18:30:43.353158 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:43.353200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:43.353371 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:43.353811 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:43.353816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:43.354352 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:43.354558 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.354572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.356083 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.356092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.357371 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.357383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.358065 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.358070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.358179 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.358750 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:43.358768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:43.361745 (+ 2977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
W20260620 18:30:43.362242 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:43.352716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52003 (local address 127.0.97.62:43487)
0620 18:30:43.352918 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:43.352923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:43.353318 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:30:43.353535 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:43.353562 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:43.353675 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:43.353780 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:43.354707 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.355895 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.357503 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.357906 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.358037 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.361982 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52003: BlockingRecv error: recv got EOF from 127.0.97.2:52003 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:43.805799 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:43.797728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:43533)
0620 18:30:43.798254 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:43.798272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:43.798392 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:43.798685 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:43.798688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:43.799074 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:43.799233 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.799242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.800349 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.800354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.801329 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.801341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.802066 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.802071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.802184 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.802840 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:43.802857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:43.805576 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
W20260620 18:30:43.806083 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:43.797822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43533 (local address 127.0.97.62:43487)
0620 18:30:43.798043 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:43.798050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:43.798367 (+ 317us) server_negotiation.cc:407] Connection header received
0620 18:30:43.798497 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:43.798500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:43.798561 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:43.798641 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:43.799399 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.800195 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.801477 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.801906 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.802005 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.805802 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43533: BlockingRecv error: recv got EOF from 127.0.97.3:43533 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:43 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:43.909130 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:43.900618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:59739)
0620 18:30:43.901211 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:43.901237 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:43.901404 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:43.901761 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:43.901766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:43.902200 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:43.902373 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.902382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.903408 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.903415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.904410 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:43.904421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.905113 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.905119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:43.905298 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.905936 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:43.905952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:43.908742 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
W20260620 18:30:43.909207 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:43.900814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59739 (local address 127.0.97.62:43487)
0620 18:30:43.901173 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:43.901180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:43.901331 (+ 151us) server_negotiation.cc:407] Connection header received
0620 18:30:43.901531 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:43.901537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:43.901621 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:43.901729 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:43.902517 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.903258 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.904547 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:43.904968 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:43.905102 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:43.908947 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59739: BlockingRecv error: recv got EOF from 127.0.97.1:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":230}
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:44.371309 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:44.362930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:43315)
0620 18:30:44.363421 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:44.363437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:44.363608 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:44.363998 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:44.364003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:44.364409 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:44.364576 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.364586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.365598 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.365603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.366517 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.366528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.367266 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.367273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.367448 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.368163 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:44.368179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:44.370971 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
W20260620 18:30:44.371393 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:44.363016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43315 (local address 127.0.97.62:43487)
0620 18:30:44.363249 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:44.363255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:44.363539 (+ 284us) server_negotiation.cc:407] Connection header received
0620 18:30:44.363798 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:44.363802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:44.363862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:44.363965 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:44.364751 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.365454 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.366664 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.367111 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.367248 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.371168 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43315: BlockingRecv error: recv got EOF from 127.0.97.2:43315 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:44.815616 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:44.806908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.3:33355)
0620 18:30:44.807477 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:44.807493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:44.807630 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:44.807993 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:44.807997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:44.808383 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:44.808550 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.808558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.809859 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.809865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.810856 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.810867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.811600 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.811606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.811778 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.812430 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:44.812447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:44.815319 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":309,"threads_started":1}
W20260620 18:30:44.815819 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:44.807024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33355 (local address 127.0.97.62:43487)
0620 18:30:44.807232 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:44.807237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:44.807612 (+ 375us) server_negotiation.cc:407] Connection header received
0620 18:30:44.807778 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:44.807782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:44.807844 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:44.807940 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:44.808702 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.809721 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.810994 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.811413 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.811575 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.815582 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33355: BlockingRecv error: recv got EOF from 127.0.97.3:33355 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:44 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:44.919886 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:44.910431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.1:45659)
0620 18:30:44.911002 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:44.911020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:44.911192 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:44.911610 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:44.911615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:44.912102 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:44.912317 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.912328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.913412 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.913425 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.914395 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:44.914408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.915185 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.915192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:44.915395 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.916140 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:44.916162 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:44.919624 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":323,"threads_started":1}
W20260620 18:30:44.920172 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:44.910520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45659 (local address 127.0.97.62:43487)
0620 18:30:44.910777 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:44.910787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:44.911112 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:30:44.911339 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:44.911343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:44.911427 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:44.911581 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:44.912489 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.913250 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.914553 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:44.915020 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:44.915182 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:44.919850 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45659: BlockingRecv error: recv got EOF from 127.0.97.1:45659 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 18:30:45 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:30:45 dist-test-slave-mcnm krb5kdc[8622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:30:45.381017 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:45.372348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43487 (local address 127.0.97.2:58821)
0620 18:30:45.372998 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:45.373015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:45.373122 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:45.373411 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:45.373414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:45.373867 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:45.374050 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:45.374059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.375035 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.375038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:45.376033 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:45.376044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.376942 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.376947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:45.377063 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:45.377716 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:45.377734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:45.380730 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":392,"threads_started":1}
W20260620 18:30:45.381155 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:45.372482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58821 (local address 127.0.97.62:43487)
0620 18:30:45.372645 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:45.372650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:45.373107 (+ 457us) server_negotiation.cc:407] Connection header received
0620 18:30:45.373223 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:45.373227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:45.373286 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:45.373369 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:45.374171 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.374908 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.376173 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.376802 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.376908 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:45.380920 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58821: BlockingRecv error: recv got EOF from 127.0.97.2:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:30:45.649066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9209
I20260620 18:30:45.661871 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:45.650199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.97.1:34137)
0620 18:30:45.650818 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:45.650829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:45.650863 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:30:45.650921 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:45.650926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:45.651144 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:45.651308 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:45.652829 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.653869 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.655066 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.655925 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.656029 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:45.658381 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:45.658389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:45.658403 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:45.658443 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:45.661369 (+ 2926us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:30:45.661643 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43312: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":338,"threads_started":1}
W20260620 18:30:45.661996 9669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43312: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:30:45.662133 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9209: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34137: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:30:45.662376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9209
I20260620 18:30:45.679709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9360
I20260620 18:30:45.691674 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:45.680715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.0.97.2:44949)
0620 18:30:45.681294 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:45.681302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:45.681331 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:30:45.681387 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:45.681393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:45.681610 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:45.681786 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:45.682965 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.684155 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.685360 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.685889 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.685986 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:45.688569 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:45.688577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:45.688591 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:45.688622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:45.691141 (+ 2519us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:30:45.691377 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37430: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
W20260620 18:30:45.691808 9670 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37430: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:30:45.691808 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9360: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44949: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:30:45.691937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9360
I20260620 18:30:45.708122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9491
I20260620 18:30:45.719655 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:45.709004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.0.97.3:38231)
0620 18:30:45.709581 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:45.709588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:45.709615 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:30:45.709678 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:45.709687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:45.709903 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:45.710058 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:45.711365 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.712405 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.713537 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:45.714061 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:45.714159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:45.716657 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:45.716665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:45.716678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:45.716711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:45.719112 (+ 2401us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:30:45.719370 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33822: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":300,"threads_started":1}
W20260620 18:30:45.719774 9671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33822: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:30:45.719774 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9491: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38231: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:30:45.719937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9491
I20260620 18:30:45.735730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9140
I20260620 18:30:45.736999 9202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:45.851128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9140
2026-06-20T18:30:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21021 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:45Z Disabled control of system clock
I20260620 18:30:45.927286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33197
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:30:46.247820 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:46.248111 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:46.248272 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:46.255587 9676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:46.255715 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:46.255782 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:46.255864 9676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:46.255929 9676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:46.265064 9676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:46.266356 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:46.268249 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:46.280550 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:46.280608 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:46.280517 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:46.281440 9676 server_base.cc:1034] running on GCE node
I20260620 18:30:46.282171 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:46.283449 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:46.284694 9676 hybrid_clock.cc:648] HybridClock initialized: now 1781980246284638 us; error 39 us; skew 500 ppm
I20260620 18:30:46.285076 9676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:46.287649 9676 webserver.cc:460] Webserver started at http://127.0.97.62:33485/ using document root <none> and password file <none>
I20260620 18:30:46.288408 9676 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:46.288542 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:46.288895 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:46.291942 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "3c36867dc08346079c98eb525ea46c41"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "0939351673ebe00dfff141c54f8a6765"
server_key_iv: "f570155fb96bd686450eceed0326e96a"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.292778 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c36867dc08346079c98eb525ea46c41"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "0939351673ebe00dfff141c54f8a6765"
server_key_iv: "f570155fb96bd686450eceed0326e96a"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.298103 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:46.301697 9690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:46.302987 9676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:30:46.303160 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "3c36867dc08346079c98eb525ea46c41"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "0939351673ebe00dfff141c54f8a6765"
server_key_iv: "f570155fb96bd686450eceed0326e96a"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.303356 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:46.317929 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:46.318951 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:46.319300 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:46.337756 9676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33197
I20260620 18:30:46.337764 9741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33197 every 8 connection(s)
I20260620 18:30:46.339080 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:46.343019 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:46.348958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9676
I20260620 18:30:46.349167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:46.350090 388 external_mini_cluster.cc:1428] Setting key 23131f3c59c1ca27d5db6bef65a04d4f
I20260620 18:30:46.354781 9742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41: Bootstrap starting.
I20260620 18:30:46.357930 9742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:46.358953 9742 log.cc:824] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:46.362079 9742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41: No bootstrap required, opened a new log
I20260620 18:30:46.364073 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:46.352387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.0.97.62:33197)
0620 18:30:46.353314 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:46.353324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:46.353361 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:30:46.354987 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:46.355008 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:46.355424 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:46.355845 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:46.356396 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.357644 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.358985 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.359871 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.360075 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:46.361343 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:46.361350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:46.361363 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:46.361900 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:46.361975 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:46.361995 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:46.362134 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:46.362398 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:30:46.362755 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260620 18:30:46.371404 9742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } }
I20260620 18:30:46.371852 9742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:46.371953 9742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c36867dc08346079c98eb525ea46c41, State: Initialized, Role: FOLLOWER
I20260620 18:30:46.372459 9742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } }
I20260620 18:30:46.372663 9742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:46.372762 9742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:46.372931 9742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:46.375413 9742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } }
I20260620 18:30:46.375922 9742 leader_election.cc:302] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c36867dc08346079c98eb525ea46c41; no voters:
I20260620 18:30:46.376437 9742 leader_election.cc:288] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:46.376606 9747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:46.377310 9747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [term 1 LEADER]: Becoming Leader. State: Replica: 3c36867dc08346079c98eb525ea46c41, State: Running, Role: LEADER
I20260620 18:30:46.377835 9747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } }
I20260620 18:30:46.378671 9742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:46.380445 9749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c36867dc08346079c98eb525ea46c41. Latest consensus state: current_term: 1 leader_uuid: "3c36867dc08346079c98eb525ea46c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } } }
I20260620 18:30:46.380836 9749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:46.381498 9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c36867dc08346079c98eb525ea46c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36867dc08346079c98eb525ea46c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33197 } } }
I20260620 18:30:46.381908 9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:46.383234 9753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:46.391041 9753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:46.399631 9753 catalog_manager.cc:1347] Generated new cluster ID: 93ebe3867c024d1986851e325f414a41
I20260620 18:30:46.399739 9753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:46.414973 9753 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:46.416447 9753 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:46.421976 9753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c36867dc08346079c98eb525ea46c41: Generated new TSK 0
I20260620 18:30:46.422819 9753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:30:46.437405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:30:46.773833 9766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:46.774082 9766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:46.774163 9766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:46.781374 9766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:46.781467 9766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:46.781602 9766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:46.790740 9766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:46.792027 9766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:46.793763 9766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:46.806716 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:46.806690 9774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:46.806831 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:46.807986 9766 server_base.cc:1034] running on GCE node
I20260620 18:30:46.808651 9766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:46.809677 9766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:46.810909 9766 hybrid_clock.cc:648] HybridClock initialized: now 1781980246810838 us; error 71 us; skew 500 ppm
I20260620 18:30:46.811235 9766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:46.813726 9766 webserver.cc:460] Webserver started at http://127.0.97.1:35315/ using document root <none> and password file <none>
I20260620 18:30:46.814415 9766 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:46.814543 9766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:46.814864 9766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:46.817817 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "bc47394f35a4495e8d01bb479f3344f5"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "f466d2e4f4342c7785b59cafb3392773"
server_key_iv: "df5140a5b3fd573aecf9c009d2ac44f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.818617 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bc47394f35a4495e8d01bb479f3344f5"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "f466d2e4f4342c7785b59cafb3392773"
server_key_iv: "df5140a5b3fd573aecf9c009d2ac44f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.823799 9766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:30:46.827095 9780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:46.828436 9766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:46.828655 9766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "bc47394f35a4495e8d01bb479f3344f5"
format_stamp: "Formatted at 2026-06-20 18:30:46 on dist-test-slave-mcnm"
server_key: "f466d2e4f4342c7785b59cafb3392773"
server_key_iv: "df5140a5b3fd573aecf9c009d2ac44f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:46.828869 9766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:46.840600 9766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:46.841436 9766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:46.841718 9766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:46.842701 9766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:46.844122 9766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:46.844239 9766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:46.844372 9766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:46.844460 9766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:46.877830 9766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44193
I20260620 18:30:46.877915 9892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44193 every 8 connection(s)
I20260620 18:30:46.879165 9766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:46.881865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9766
I20260620 18:30:46.882061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:46.882893 388 external_mini_cluster.cc:1428] Setting key de4cf8cede1e065daf9fb68599130d59
I20260620 18:30:46.885700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:30:46.892578 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:46.882002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35873 (local address 127.0.97.62:33197)
0620 18:30:46.882689 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:46.882697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:46.884044 (+ 1347us) server_negotiation.cc:407] Connection header received
0620 18:30:46.885977 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:46.885984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:46.886057 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:46.886189 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:46.887241 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.887997 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.889551 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.889951 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.890082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:46.891503 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:46.891533 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:46.891542 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:46.891565 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:46.891619 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:46.891638 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:46.891735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:46.892038 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:30:46.892148 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":433,"threads_started":1}
I20260620 18:30:46.893445 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:46.882720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33197 (local address 127.0.97.1:35873)
0620 18:30:46.883895 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:46.883941 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:46.885645 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:46.886414 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:46.886423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:46.886519 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:46.887065 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:46.887080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.888166 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.888170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:46.889409 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:46.889420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:46.890170 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:46.890178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:46.890429 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:46.891188 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:46.891190 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:46.891235 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:46.891311 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:46.891775 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:46.891787 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:46.891795 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:30:46.892024 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 18:30:46.892316 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":358,"threads_started":1}
I20260620 18:30:46.895047 9893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33197
I20260620 18:30:46.895342 9893 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:46.896134 9893 heartbeater.cc:511] Master 127.0.97.62:33197 requested a full tablet report, sending...
I20260620 18:30:46.898231 9707 ts_manager.cc:194] Registered new tserver with Master: bc47394f35a4495e8d01bb479f3344f5 (127.0.97.1:44193)
I20260620 18:30:46.899808 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35873
W20260620 18:30:47.214937 9897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:47.215207 9897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:47.215361 9897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:47.222577 9897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:47.222672 9897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:47.222828 9897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:47.231805 9897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:47.233139 9897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:47.234930 9897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:47.247501 9902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:47.247458 9905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:47.247490 9903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:47.248317 9897 server_base.cc:1034] running on GCE node
I20260620 18:30:47.248946 9897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:47.249856 9897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:47.251097 9897 hybrid_clock.cc:648] HybridClock initialized: now 1781980247251037 us; error 58 us; skew 500 ppm
I20260620 18:30:47.251421 9897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:47.253746 9897 webserver.cc:460] Webserver started at http://127.0.97.2:40933/ using document root <none> and password file <none>
I20260620 18:30:47.254429 9897 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:47.254559 9897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:47.254876 9897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:47.257737 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b72c2bba1324a55a6a65595e25a8f94"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "77d11fdb371d560006b1c4926c419af5"
server_key_iv: "670ae3668e8de48e49dd4f2898e4f6f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.258486 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b72c2bba1324a55a6a65595e25a8f94"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "77d11fdb371d560006b1c4926c419af5"
server_key_iv: "670ae3668e8de48e49dd4f2898e4f6f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.263357 9897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:30:47.266685 9911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.267997 9897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:47.268186 9897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "8b72c2bba1324a55a6a65595e25a8f94"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "77d11fdb371d560006b1c4926c419af5"
server_key_iv: "670ae3668e8de48e49dd4f2898e4f6f7"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.268393 9897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:47.286907 9897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:47.287771 9897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:47.288039 9897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:47.288926 9897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:47.290352 9897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:47.290483 9897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.290619 9897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:47.290709 9897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.323995 9897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36687
I20260620 18:30:47.324041 10023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36687 every 8 connection(s)
I20260620 18:30:47.325289 9897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:47.328570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 9897
I20260620 18:30:47.328747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:47.329202 388 external_mini_cluster.cc:1428] Setting key 5dfb35f11d377c2a2c9beeb8466bb0df
I20260620 18:30:47.331782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:30:47.339955 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.327967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41487 (local address 127.0.97.62:33197)
0620 18:30:47.328173 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:47.328179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:47.329757 (+ 1578us) server_negotiation.cc:407] Connection header received
0620 18:30:47.331743 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:47.331748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:47.331850 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:47.331967 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:47.333547 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.334358 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.336823 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.337289 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.337429 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.338795 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:47.338805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:47.338828 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:47.338853 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:47.338889 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:47.338905 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:47.339009 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:47.339347 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:30:47.339452 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:30:47.341943 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.328513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33197 (local address 127.0.97.2:41487)
0620 18:30:47.329604 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:47.329654 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:47.331421 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:47.332295 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:47.332309 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:47.332444 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:47.333335 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:47.333364 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.334521 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.334529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:47.336659 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:47.336677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.337556 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.337571 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:47.337773 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.338455 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:47.338458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:47.338532 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:47.338612 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:47.339079 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:47.339088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:47.339096 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:30:47.340260 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0620 18:30:47.340668 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":383,"threads_started":1}
I20260620 18:30:47.343338 10024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33197
I20260620 18:30:47.343665 10024 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:47.344524 10024 heartbeater.cc:511] Master 127.0.97.62:33197 requested a full tablet report, sending...
I20260620 18:30:47.346012 9707 ts_manager.cc:194] Registered new tserver with Master: 8b72c2bba1324a55a6a65595e25a8f94 (127.0.97.2:36687)
I20260620 18:30:47.347047 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41487
W20260620 18:30:47.651589 10027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:47.651892 10027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:47.652051 10027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:47.659034 10027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:47.659180 10027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:47.659349 10027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:47.667932 10027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:47.669162 10027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:47.670890 10027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:47.684625 10032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:47.684789 10033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:47.684996 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:47.685782 10027 server_base.cc:1034] running on GCE node
I20260620 18:30:47.686441 10027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:47.687424 10027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:47.688650 10027 hybrid_clock.cc:648] HybridClock initialized: now 1781980247688599 us; error 44 us; skew 500 ppm
I20260620 18:30:47.688998 10027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:47.695890 10027 webserver.cc:460] Webserver started at http://127.0.97.3:36705/ using document root <none> and password file <none>
I20260620 18:30:47.696640 10027 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:47.696731 10027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:47.697096 10027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:47.700103 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a4d8dba6d2d44d1aa604145002a7e47"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "097d9cbbde833bc4f4e8d56377c12744"
server_key_iv: "17a32ce0c10519334c5af33da946f879"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.700902 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a4d8dba6d2d44d1aa604145002a7e47"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "097d9cbbde833bc4f4e8d56377c12744"
server_key_iv: "17a32ce0c10519334c5af33da946f879"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.706156 10027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:47.709702 10041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.711050 10027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:30:47.711216 10027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "7a4d8dba6d2d44d1aa604145002a7e47"
format_stamp: "Formatted at 2026-06-20 18:30:47 on dist-test-slave-mcnm"
server_key: "097d9cbbde833bc4f4e8d56377c12744"
server_key_iv: "17a32ce0c10519334c5af33da946f879"
server_key_version: "encryptionkey@0"
I20260620 18:30:47.711406 10027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:47.726109 10027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:47.726892 10027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:47.727170 10027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:47.728168 10027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:47.729585 10027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:47.729671 10027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.729822 10027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:47.729893 10027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:47.764145 10027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33329
I20260620 18:30:47.764248 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33329 every 8 connection(s)
I20260620 18:30:47.765506 10027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:47.766520 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10027
I20260620 18:30:47.766695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:47.767232 388 external_mini_cluster.cc:1428] Setting key 2357b691f4a911eedec2ff495deb0d6e
I20260620 18:30:47.778342 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.768369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55951 (local address 127.0.97.62:33197)
0620 18:30:47.768755 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:47.768761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:47.770330 (+ 1569us) server_negotiation.cc:407] Connection header received
0620 18:30:47.772238 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:47.772243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:47.772349 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:47.772452 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:47.773460 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.774170 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.775772 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.776187 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.776323 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.777520 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:47.777526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:47.777534 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:47.777554 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:47.777595 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:47.777607 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:47.777694 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:47.777904 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:30:47.778001 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 18:30:47.779378 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.769109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33197 (local address 127.0.97.3:55951)
0620 18:30:47.770188 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:47.770238 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:47.771995 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:47.772683 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:47.772691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:47.772772 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:47.773304 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:47.773318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.774321 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.774333 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:47.775642 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:47.775653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.776396 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.776402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:47.776626 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.777242 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:47.777244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:47.777279 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:47.777345 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:47.777705 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:47.777717 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:47.777730 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:30:47.777895 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 18:30:47.778245 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":354,"threads_started":1}
I20260620 18:30:47.780714 10154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33197
I20260620 18:30:47.781023 10154 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:47.781837 10154 heartbeater.cc:511] Master 127.0.97.62:33197 requested a full tablet report, sending...
I20260620 18:30:47.783138 9707 ts_manager.cc:194] Registered new tserver with Master: 7a4d8dba6d2d44d1aa604145002a7e47 (127.0.97.3:33329)
I20260620 18:30:47.783977 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:55951
I20260620 18:30:47.794934 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:30:47.801100 9895 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:47.799411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.97.62:33197)
0620 18:30:47.799667 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:47.799676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:47.800010 (+ 334us) server_negotiation.cc:407] Connection header received
0620 18:30:47.800136 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:47.800140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:47.800215 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:47.800309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:47.800872 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56810: BlockingRecv error: recv got EOF from 127.0.0.1:56810 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:30:47.802129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9766
I20260620 18:30:47.809990 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.803228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.0.97.1:44193)
0620 18:30:47.803787 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:47.803794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:47.803819 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:30:47.803871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:47.803876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:47.804102 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:47.804223 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:47.804627 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.805595 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.806916 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.807435 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.807603 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.808627 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:47.808634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:47.808646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:47.809215 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:47.809258 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:47.809276 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:47.809361 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:47.809509 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:30:47.809608 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":309,"threads_started":1}
I20260620 18:30:47.811223 9888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:47.895015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9766
I20260620 18:30:47.912439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9897
I20260620 18:30:47.920405 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:47.913347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.0.97.2:36687)
0620 18:30:47.913861 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:47.913868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:47.913895 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:30:47.913962 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:47.913968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:47.914178 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:47.914313 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:47.914738 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.915841 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.917248 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:47.917785 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:47.917884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:47.918881 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:47.918899 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:47.918912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:47.919604 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:47.919649 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:47.919674 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:47.919767 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:47.919906 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:30:47.920060 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
I20260620 18:30:47.921617 10019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:48.002665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9897
I20260620 18:30:48.019302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10027
I20260620 18:30:48.027699 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:48.020443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.0.97.3:33329)
0620 18:30:48.021166 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:48.021174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:48.021204 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:30:48.021266 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:48.021273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:48.021489 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:48.021626 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:48.022023 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:48.023076 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:48.024382 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:48.024950 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:48.025074 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:48.026104 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:48.026114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:48.026135 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:48.026705 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:48.026756 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:48.026774 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:48.026863 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:48.027098 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:30:48.027256 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":402,"threads_started":1}
I20260620 18:30:48.028947 10149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:48.111251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10027
I20260620 18:30:48.127117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 9676
I20260620 18:30:48.128423 9737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:48.228116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 9676
2026-06-20T18:30:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2371 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:48Z Disabled control of system clock
I20260620 18:30:48.297822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44825
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:30:48.615378 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:48.615731 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:48.615907 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:48.623140 10173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:48.623242 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:48.623330 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:48.623440 10173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:48.623545 10173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:48.633221 10173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:48.634500 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:48.636315 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:48.648080 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:48.648041 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:48.648121 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:48.648823 10173 server_base.cc:1034] running on GCE node
I20260620 18:30:48.649538 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:48.650820 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:48.652058 10173 hybrid_clock.cc:648] HybridClock initialized: now 1781980248652002 us; error 46 us; skew 500 ppm
I20260620 18:30:48.652371 10173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:48.654544 10173 webserver.cc:460] Webserver started at http://127.0.97.62:42543/ using document root <none> and password file <none>
I20260620 18:30:48.655169 10173 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:48.655251 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:48.655642 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:48.658337 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "349f541c6fdc4d359f26e75927290b38"
format_stamp: "Formatted at 2026-06-20 18:30:48 on dist-test-slave-mcnm"
server_key: "ad605d7a91ac84f97b51261d51bcb9d6"
server_key_iv: "871000683c5a7db3ca14a2a511525a3f"
server_key_version: "encryptionkey@0"
I20260620 18:30:48.659076 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "349f541c6fdc4d359f26e75927290b38"
format_stamp: "Formatted at 2026-06-20 18:30:48 on dist-test-slave-mcnm"
server_key: "ad605d7a91ac84f97b51261d51bcb9d6"
server_key_iv: "871000683c5a7db3ca14a2a511525a3f"
server_key_version: "encryptionkey@0"
I20260620 18:30:48.663870 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260620 18:30:48.667102 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:48.668488 10173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:48.668622 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "349f541c6fdc4d359f26e75927290b38"
format_stamp: "Formatted at 2026-06-20 18:30:48 on dist-test-slave-mcnm"
server_key: "ad605d7a91ac84f97b51261d51bcb9d6"
server_key_iv: "871000683c5a7db3ca14a2a511525a3f"
server_key_version: "encryptionkey@0"
I20260620 18:30:48.668787 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:48.674103 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:48.674774 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:48.675012 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:48.692986 10173 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44825
I20260620 18:30:48.693032 10238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44825 every 8 connection(s)
I20260620 18:30:48.694278 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:48.696440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10173
I20260620 18:30:48.696651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:48.697443 388 external_mini_cluster.cc:1428] Setting key 874a7750bb86aed3517b0c377b9693fc
I20260620 18:30:48.698746 10239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:48.705621 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:48.699476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.0.97.62:44825)
0620 18:30:48.700514 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:48.700524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:48.700563 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:30:48.701973 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:48.701993 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:48.702390 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:48.702733 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:48.703887 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:48.703894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:48.703906 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:48.703921 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:48.704022 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:48.704040 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:48.704240 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:48.704386 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:30:48.704679 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":371,"threads_started":1}
I20260620 18:30:48.710840 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38: Bootstrap starting.
I20260620 18:30:48.713750 10239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:48.714707 10239 log.cc:824] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:48.717582 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38: No bootstrap required, opened a new log
I20260620 18:30:48.733352 10239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260620 18:30:48.733964 10239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:48.734084 10239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349f541c6fdc4d359f26e75927290b38, State: Initialized, Role: FOLLOWER
I20260620 18:30:48.735004 10239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260620 18:30:48.735313 10239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:48.735471 10239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:48.735706 10239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:48.739682 10239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260620 18:30:48.740396 10239 leader_election.cc:302] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 349f541c6fdc4d359f26e75927290b38; no voters:
I20260620 18:30:48.741164 10239 leader_election.cc:288] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:48.741253 10244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:48.742604 10244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [term 1 LEADER]: Becoming Leader. State: Replica: 349f541c6fdc4d359f26e75927290b38, State: Running, Role: LEADER
I20260620 18:30:48.743194 10244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260620 18:30:48.743741 10239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:48.747208 10246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 349f541c6fdc4d359f26e75927290b38. Latest consensus state: current_term: 1 leader_uuid: "349f541c6fdc4d359f26e75927290b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } } }
I20260620 18:30:48.747423 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "349f541c6fdc4d359f26e75927290b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349f541c6fdc4d359f26e75927290b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } } }
I20260620 18:30:48.747759 10246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:48.747839 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:48.759647 10252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:48.776125 10252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:48.800287 10252 catalog_manager.cc:1347] Generated new cluster ID: 7d8bb82a99664649b7c3026a1a4b9aa2
I20260620 18:30:48.800434 10252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:48.816952 10262 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:30:48.854878 10252 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:48.855305 10252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:48.866263 10252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 349f541c6fdc4d359f26e75927290b38: Generated new TSK 0
I20260620 18:30:48.867056 10252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:30:48.881825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--builtin_ntp_servers=127.0.97.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:30:49.258935 10263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:49.259253 10263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:49.259405 10263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:49.266674 10263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:49.266769 10263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:49.266932 10263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:49.276283 10263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:49.277493 10263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:49.279229 10263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:49.292512 10268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:49.292454 10271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:49.292428 10269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:49.293527 10263 server_base.cc:1034] running on GCE node
I20260620 18:30:49.294098 10263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:49.295047 10263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:49.296317 10263 hybrid_clock.cc:648] HybridClock initialized: now 1781980249296245 us; error 65 us; skew 500 ppm
I20260620 18:30:49.296648 10263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:49.299019 10263 webserver.cc:460] Webserver started at http://127.0.97.1:33829/ using document root <none> and password file <none>
I20260620 18:30:49.299768 10263 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:49.299897 10263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:49.300210 10263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:49.304695 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "134319f9c8a440a9ac6bf192eef29fd8"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "e7519f59af4ba23cbb6aa50b80a048ae"
server_key_iv: "0097b7b33482bd1768ccbc194992db9c"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.305490 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "134319f9c8a440a9ac6bf192eef29fd8"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "e7519f59af4ba23cbb6aa50b80a048ae"
server_key_iv: "0097b7b33482bd1768ccbc194992db9c"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.310643 10263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:30:49.314245 10277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.315655 10263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:49.315846 10263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "134319f9c8a440a9ac6bf192eef29fd8"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "e7519f59af4ba23cbb6aa50b80a048ae"
server_key_iv: "0097b7b33482bd1768ccbc194992db9c"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.316076 10263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:49.321762 10263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:49.322513 10263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:49.322778 10263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:49.323738 10263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:49.325090 10263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:49.325209 10263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.325344 10263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:49.325429 10263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.358235 10263 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41073
I20260620 18:30:49.358285 10389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41073 every 8 connection(s)
I20260620 18:30:49.359644 10263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:49.364564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10263
I20260620 18:30:49.364770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:49.365491 388 external_mini_cluster.cc:1428] Setting key cd7bb5738561881691408f21aa8a6284
I20260620 18:30:49.368285 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:49.362219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40381 (local address 127.0.97.62:44825)
0620 18:30:49.362685 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:49.362693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:49.363905 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 18:30:49.365744 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:49.365750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:49.365828 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:49.365986 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:49.367421 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:49.367425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:49.367428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:49.367443 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:49.367522 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:49.367572 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:49.367702 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:49.367875 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 18:30:49.367935 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":208,"threads_started":1}
I20260620 18:30:49.368664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--builtin_ntp_servers=127.0.97.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:30:49.369222 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:49.362772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.1:40381)
0620 18:30:49.363744 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:49.363801 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:49.365444 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:49.366227 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:49.366238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:49.366370 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:49.367087 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:49.367089 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:49.367126 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:49.367211 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:49.367665 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:49.367673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:49.367678 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:30:49.367906 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 18:30:49.368106 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260620 18:30:49.370633 10390 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260620 18:30:49.370936 10390 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:49.371631 10390 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260620 18:30:49.373837 10204 ts_manager.cc:194] Registered new tserver with Master: 134319f9c8a440a9ac6bf192eef29fd8 (127.0.97.1:41073)
W20260620 18:30:49.690987 10394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:49.691272 10394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:49.691413 10394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:49.698742 10394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:49.698849 10394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:49.699045 10394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:49.708336 10394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:49.709635 10394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:49.711436 10394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:49.725176 10399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:49.725090 10400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:49.725147 10402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:49.725876 10394 server_base.cc:1034] running on GCE node
I20260620 18:30:49.726500 10394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:49.727504 10394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:49.728817 10394 hybrid_clock.cc:648] HybridClock initialized: now 1781980249728766 us; error 45 us; skew 500 ppm
I20260620 18:30:49.729176 10394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:49.731750 10394 webserver.cc:460] Webserver started at http://127.0.97.2:40677/ using document root <none> and password file <none>
I20260620 18:30:49.732497 10394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:49.732630 10394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:49.732951 10394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:49.735942 10394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "9521689ee13e47a08f83de2a504821c9"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "622d6be595545076e9be045bf9128ca9"
server_key_iv: "11cfc69e82f956b3ea637dd335367f4f"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.736747 10394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9521689ee13e47a08f83de2a504821c9"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "622d6be595545076e9be045bf9128ca9"
server_key_iv: "11cfc69e82f956b3ea637dd335367f4f"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.742071 10394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 18:30:49.745819 10408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.747175 10394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:30:49.747378 10394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "9521689ee13e47a08f83de2a504821c9"
format_stamp: "Formatted at 2026-06-20 18:30:49 on dist-test-slave-mcnm"
server_key: "622d6be595545076e9be045bf9128ca9"
server_key_iv: "11cfc69e82f956b3ea637dd335367f4f"
server_key_version: "encryptionkey@0"
I20260620 18:30:49.747674 10394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:49.753549 10394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:49.754371 10394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:49.754645 10394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:49.755630 10394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:49.757009 10394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:49.757125 10394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.757256 10394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:49.757359 10394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:49.791461 10394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37021
I20260620 18:30:49.791582 10520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37021 every 8 connection(s)
I20260620 18:30:49.792886 10394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:49.801416 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:49.795488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49793 (local address 127.0.97.62:44825)
0620 18:30:49.795720 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:49.795726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:49.797126 (+ 1400us) server_negotiation.cc:407] Connection header received
0620 18:30:49.799083 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:49.799088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:49.799156 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:49.799246 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:49.800705 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:49.800710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:49.800713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:49.800731 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:49.800763 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:49.800774 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:49.800907 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:49.801133 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:30:49.801187 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:30:49.802426 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:49.796018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.2:49793)
0620 18:30:49.796959 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:49.796996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:49.798762 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:49.799470 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:49.799479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:49.799655 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:49.800385 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:49.800387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:49.800429 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:49.800517 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:49.800912 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:49.800920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:49.800927 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:30:49.801173 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 18:30:49.801427 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260620 18:30:49.802934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10394
I20260620 18:30:49.803129 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:49.803721 388 external_mini_cluster.cc:1428] Setting key 480741cfbf7e7a5cc3942e71d338a683
I20260620 18:30:49.804170 10521 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260620 18:30:49.804600 10521 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:49.805198 10521 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260620 18:30:49.806699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--builtin_ntp_servers=127.0.97.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:30:49.806733 10204 ts_manager.cc:194] Registered new tserver with Master: 9521689ee13e47a08f83de2a504821c9 (127.0.97.2:37021)
W20260620 18:30:50.135087 10524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:50.135394 10524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:50.135617 10524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:50.143091 10524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:50.143201 10524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:50.143431 10524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:50.152990 10524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:50.154294 10524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:50.156147 10524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:50.169348 10529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:50.169320 10530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:50.169556 10532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:50.170017 10524 server_base.cc:1034] running on GCE node
I20260620 18:30:50.170627 10524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:50.171715 10524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:50.172932 10524 hybrid_clock.cc:648] HybridClock initialized: now 1781980250172895 us; error 37 us; skew 500 ppm
I20260620 18:30:50.173242 10524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:30:50.175580 10524 webserver.cc:460] Webserver started at http://127.0.97.3:38809/ using document root <none> and password file <none>
I20260620 18:30:50.176240 10524 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:50.176373 10524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:50.176684 10524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:50.179628 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ac1c6af80f24adc9477e0e846422002"
format_stamp: "Formatted at 2026-06-20 18:30:50 on dist-test-slave-mcnm"
server_key: "88dd6deac3e722373bc8ad733731def5"
server_key_iv: "15c9a1d297f40f41aa766c284328c497"
server_key_version: "encryptionkey@0"
I20260620 18:30:50.180652 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ac1c6af80f24adc9477e0e846422002"
format_stamp: "Formatted at 2026-06-20 18:30:50 on dist-test-slave-mcnm"
server_key: "88dd6deac3e722373bc8ad733731def5"
server_key_iv: "15c9a1d297f40f41aa766c284328c497"
server_key_version: "encryptionkey@0"
I20260620 18:30:50.185711 10524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:30:50.189132 10538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:50.190603 10524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:30:50.190791 10524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "9ac1c6af80f24adc9477e0e846422002"
format_stamp: "Formatted at 2026-06-20 18:30:50 on dist-test-slave-mcnm"
server_key: "88dd6deac3e722373bc8ad733731def5"
server_key_iv: "15c9a1d297f40f41aa766c284328c497"
server_key_version: "encryptionkey@0"
I20260620 18:30:50.191026 10524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:50.196503 10524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:50.197261 10524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:50.197523 10524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:50.198424 10524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:50.199862 10524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:50.199978 10524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:50.200111 10524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:50.200199 10524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:50.233403 10524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42405
I20260620 18:30:50.233491 10650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42405 every 8 connection(s)
I20260620 18:30:50.234715 10524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:50.239646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10524
I20260620 18:30:50.239851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:50.240278 388 external_mini_cluster.cc:1428] Setting key a2f747c0e9cd081d11e287591d1bf4df
I20260620 18:30:50.243584 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:50.237298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47817 (local address 127.0.97.62:44825)
0620 18:30:50.237503 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:50.237508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:50.239010 (+ 1502us) server_negotiation.cc:407] Connection header received
0620 18:30:50.241044 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:50.241048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:50.241108 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:50.241189 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:50.242729 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:50.242733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:50.242737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:50.242754 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:50.242786 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:50.242799 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:50.242899 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:50.243140 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:30:50.243207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 18:30:50.244462 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:50.237805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.3:47817)
0620 18:30:50.238802 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:50.238851 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:50.240730 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:50.241464 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:50.241478 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:50.241611 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:50.242381 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:30:50.242385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:50.242441 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:30:50.242554 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:50.242908 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:50.242918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:50.242925 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:30:50.243231 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 18:30:50.243549 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260620 18:30:50.245836 10651 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260620 18:30:50.246129 10651 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:50.246742 10651 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260620 18:30:50.248095 10204 ts_manager.cc:194] Registered new tserver with Master: 9ac1c6af80f24adc9477e0e846422002 (127.0.97.3:42405)
I20260620 18:30:50.255712 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:30:50.261691 10392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:30:50.260125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.97.62:44825)
0620 18:30:50.260284 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:50.260288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:50.260769 (+ 481us) server_negotiation.cc:407] Connection header received
0620 18:30:50.260961 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:50.260967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:50.261026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:50.261138 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:50.261442 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46020: BlockingRecv error: recv got EOF from 127.0.0.1:46020 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:30:50.262127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10263
I20260620 18:30:50.266032 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:50.263495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.97.1:41073)
0620 18:30:50.264081 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:50.264088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:50.264111 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:30:50.264164 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:50.264169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:50.264367 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:50.264485 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:50.265400 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:50.265406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:50.265417 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:50.265436 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:50.265475 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:50.265488 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:50.265589 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:50.265734 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:30:50.265808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260620 18:30:50.267180 10385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:50.347507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10263
I20260620 18:30:50.364733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10394
I20260620 18:30:50.368546 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:50.365772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.97.2:37021)
0620 18:30:50.366347 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:50.366354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:50.366380 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:30:50.366440 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:50.366446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:50.366657 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:50.366788 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:50.367741 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:50.367750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:50.367771 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:50.367790 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:50.367831 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:50.367846 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:50.367953 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:50.368135 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:30:50.368288 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":296,"threads_started":1}
I20260620 18:30:50.369850 10516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:50.451278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10394
I20260620 18:30:50.468293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10524
I20260620 18:30:50.472478 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:50.469221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.0.97.3:42405)
0620 18:30:50.469874 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:50.469881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:50.469911 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:30:50.469972 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:50.469979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:50.470241 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:50.470423 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:50.471632 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:50.471638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:50.471653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:30:50.471670 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:50.471710 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:30:50.471724 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:50.471908 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:50.471981 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:30:50.472081 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":352,"threads_started":1}
I20260620 18:30:50.473839 10646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:50.554795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10524
I20260620 18:30:50.570307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10173
I20260620 18:30:50.571501 10234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:50.672533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10173
2026-06-20T18:30:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2444 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:50 dist-test-slave-mcnm krb5kdc[10669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:50 dist-test-slave-mcnm krb5kdc[10669](info): set up 2 sockets
Jun 20 18:30:50 dist-test-slave-mcnm krb5kdc[10669](info): commencing operation
krb5kdc: starting...
W20260620 18:30:51.776402 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:51 dist-test-slave-mcnm krb5kdc[10669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:30:52.019495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:52.333694 10682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:52.333959 10682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:52.334117 10682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:52.341286 10682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:30:52.341380 10682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:52.341462 10682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:52.341562 10682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:30:52.341645 10682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:30:52.351156 10682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:52.352420 10682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:52.353695 10682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:52.365342 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:52.365463 10690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:52.365326 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:52.366250 10682 server_base.cc:1034] running on GCE node
I20260620 18:30:52.367065 10682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:52.368645 10682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:52.369889 10682 hybrid_clock.cc:648] HybridClock initialized: now 1781980252369854 us; error 53 us; skew 500 ppm
I20260620 18:30:52.370245 10682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:52 dist-test-slave-mcnm krb5kdc[10669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:52.373340 10682 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:30:52.374837 10682 webserver.cc:460] Webserver started at http://127.0.97.62:44263/ using document root <none> and password file <none>
I20260620 18:30:52.375458 10682 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:52.375627 10682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:52.375949 10682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:52.378881 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "6bd1e05c00db4661b6d6ada36637fbb8"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "43783de1138a481bdc3ae8ab78c202ee"
server_key_iv: "31e375738e1d2d105b0569c9d912ee80"
server_key_version: "encryptionkey@0"
I20260620 18:30:52.379698 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "6bd1e05c00db4661b6d6ada36637fbb8"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "43783de1138a481bdc3ae8ab78c202ee"
server_key_iv: "31e375738e1d2d105b0569c9d912ee80"
server_key_version: "encryptionkey@0"
I20260620 18:30:52.384713 10682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:30:52.388810 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:52.390347 10682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:30:52.390491 10682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "6bd1e05c00db4661b6d6ada36637fbb8"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "43783de1138a481bdc3ae8ab78c202ee"
server_key_iv: "31e375738e1d2d105b0569c9d912ee80"
server_key_version: "encryptionkey@0"
I20260620 18:30:52.390650 10682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:52.413884 10682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:52.417841 10682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:52.418072 10682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:52.436743 10682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36503
I20260620 18:30:52.436789 10748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36503 every 8 connection(s)
I20260620 18:30:52.438035 10682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:30:52.440203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10682
I20260620 18:30:52.440451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:30:52.441216 388 external_mini_cluster.cc:1428] Setting key 695217cb39a06231f610c28152e828c4
I20260620 18:30:52.442330 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:30:52 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:52.455264 10749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: Bootstrap starting.
I20260620 18:30:52.458856 10749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:52.460049 10749 log.cc:824] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:52.463021 10749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: No bootstrap required, opened a new log
I20260620 18:30:52.463958 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:52.443161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.97.62:36503)
0620 18:30:52.443970 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:52.443988 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:52.444031 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:30:52.445624 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:52.445645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:52.446034 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:52.446459 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:52.447974 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:52.449384 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:52.450726 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:52.451435 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:52.451725 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:52.456128 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:52.456136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:52.456150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:52.456183 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:52.459658 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:52.460413 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:52.460422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:52.460438 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:52.460621 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:52.461042 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:52.461049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:52.461054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:52.461600 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:52.461889 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:52.462384 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:30:52.462784 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":401,"threads_started":1}
I20260620 18:30:52.472222 10749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } }
I20260620 18:30:52.472604 10749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:52.472723 10749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd1e05c00db4661b6d6ada36637fbb8, State: Initialized, Role: FOLLOWER
I20260620 18:30:52.473331 10749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } }
I20260620 18:30:52.473635 10749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:30:52.473791 10749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:30:52.473985 10749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:52.476699 10749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } }
I20260620 18:30:52.477231 10749 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd1e05c00db4661b6d6ada36637fbb8; no voters:
I20260620 18:30:52.477869 10749 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:30:52.477969 10754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:52.478735 10754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd1e05c00db4661b6d6ada36637fbb8, State: Running, Role: LEADER
I20260620 18:30:52.479333 10754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } }
I20260620 18:30:52.480283 10749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:30:52.482275 10756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd1e05c00db4661b6d6ada36637fbb8. Latest consensus state: current_term: 1 leader_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } } }
I20260620 18:30:52.482618 10756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:52.483093 10755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd1e05c00db4661b6d6ada36637fbb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36503 } } }
I20260620 18:30:52.483358 10755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:30:52.488955 10761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:30:52.494797 10761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:30:52.503623 10761 catalog_manager.cc:1347] Generated new cluster ID: 51f8521db94c44bc8cae30fb25a0a935
I20260620 18:30:52.503756 10761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:30:52.530640 10761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:30:52.532543 10761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:30:52.554845 10761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: Generated new TSK 0
I20260620 18:30:52.555816 10761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:30:52.614684 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:52.946218 10775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:52.946514 10775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:52.946609 10775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:52.953702 10775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:52.953823 10775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:52.953935 10775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:30:52.964289 10775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:52.965641 10775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:52.966874 10775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:52.979586 10781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:52.979563 10783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:52.979628 10780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:52.985384 10775 server_base.cc:1034] running on GCE node
I20260620 18:30:52.986114 10775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:52.987102 10775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:52.988318 10775 hybrid_clock.cc:648] HybridClock initialized: now 1781980252988271 us; error 39 us; skew 500 ppm
I20260620 18:30:52.988680 10775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:52 dist-test-slave-mcnm krb5kdc[10669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:52.995909 10775 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:30:52.997498 10775 webserver.cc:460] Webserver started at http://127.0.97.1:39201/ using document root <none> and password file <none>
I20260620 18:30:52.998230 10775 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:52.998409 10775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:52.998729 10775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:53.001703 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "33c6e0776a26461f81fe15752dc13e12"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "8b905f37027a587237c22de3ba143184"
server_key_iv: "cc184fb795b43e7061cdb77b0b8ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.002458 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33c6e0776a26461f81fe15752dc13e12"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "8b905f37027a587237c22de3ba143184"
server_key_iv: "cc184fb795b43e7061cdb77b0b8ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.007905 10775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:30:53.011360 10790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.012792 10775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:30:53.012998 10775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "33c6e0776a26461f81fe15752dc13e12"
format_stamp: "Formatted at 2026-06-20 18:30:52 on dist-test-slave-mcnm"
server_key: "8b905f37027a587237c22de3ba143184"
server_key_iv: "cc184fb795b43e7061cdb77b0b8ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.013218 10775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:53.035181 10775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:53.039438 10775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:53.039775 10775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:53.040766 10775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:53.042120 10775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:53.042227 10775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.042375 10775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:53.042449 10775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.076570 10775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41771
I20260620 18:30:53.076664 10902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41771 every 8 connection(s)
I20260620 18:30:53.077840 10775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:30:53.087108 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10775
I20260620 18:30:53.087339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:30:53.087996 388 external_mini_cluster.cc:1428] Setting key a1ba751d285072581de807c9903e1bae
Jun 20 18:30:53 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:53.098694 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:53.080545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48311 (local address 127.0.97.62:36503)
0620 18:30:53.081028 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:53.081033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:53.082025 (+ 992us) server_negotiation.cc:407] Connection header received
0620 18:30:53.083896 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:53.083913 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:53.083996 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:53.084111 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:53.085568 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.086261 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.088014 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.088460 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.088590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:53.092468 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:53.092474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:53.092479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:53.092518 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:53.095441 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:53.096254 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:53.096274 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:53.096279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:53.096403 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:53.096861 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:53.096873 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:53.096876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:53.097162 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:53.097326 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:53.098040 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 18:30:53.098254 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":240,"threads_started":1}
I20260620 18:30:53.099731 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:53.081028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36503 (local address 127.0.97.1:48311)
0620 18:30:53.081854 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:53.081891 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:53.083623 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:53.084343 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:53.084351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:53.084842 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:53.085382 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:53.085413 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.086424 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.086428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:53.087857 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:53.087869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.088671 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.088679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:53.088961 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:53.089710 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:53.089731 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:53.092255 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:53.095640 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:53.095650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:53.095665 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:53.096108 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:53.096521 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:53.096528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:53.096532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:53.096734 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:53.097299 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:53.097307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:53.097644 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 18:30:53.097919 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:30:53.098438 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":326,"threads_started":1}
I20260620 18:30:53.101414 10903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36503
I20260620 18:30:53.101738 10903 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:53.102443 10903 heartbeater.cc:511] Master 127.0.97.62:36503 requested a full tablet report, sending...
I20260620 18:30:53.104784 10714 ts_manager.cc:194] Registered new tserver with Master: 33c6e0776a26461f81fe15752dc13e12 (127.0.97.1:41771)
I20260620 18:30:53.106989 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48311
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:30:53.140134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:53.456688 10909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:53.456959 10909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:53.457113 10909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:53.464210 10909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:53.464324 10909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:53.464505 10909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:30:53.473743 10909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:53.474913 10909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:53.476186 10909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:53.488700 10917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:53.488915 10915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:53.489563 10914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:53.489539 10909 server_base.cc:1034] running on GCE node
I20260620 18:30:53.490249 10909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:53.491204 10909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:53.492453 10909 hybrid_clock.cc:648] HybridClock initialized: now 1781980253492395 us; error 51 us; skew 500 ppm
I20260620 18:30:53.492774 10909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:53 dist-test-slave-mcnm krb5kdc[10669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:53.495988 10909 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:30:53.497284 10909 webserver.cc:460] Webserver started at http://127.0.97.2:32975/ using document root <none> and password file <none>
I20260620 18:30:53.497943 10909 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:53.498081 10909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:53.498363 10909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:53.501175 10909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "d36511a534054e19a7777f1eeabd4da9"
format_stamp: "Formatted at 2026-06-20 18:30:53 on dist-test-slave-mcnm"
server_key: "cf6430bc802b5480aad4d0b15b10f88f"
server_key_iv: "9b4fbc6f64d6c99e6289fe51c34251e4"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.501894 10909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d36511a534054e19a7777f1eeabd4da9"
format_stamp: "Formatted at 2026-06-20 18:30:53 on dist-test-slave-mcnm"
server_key: "cf6430bc802b5480aad4d0b15b10f88f"
server_key_iv: "9b4fbc6f64d6c99e6289fe51c34251e4"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.506860 10909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:30:53.510366 10924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.512003 10909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:30:53.512209 10909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "d36511a534054e19a7777f1eeabd4da9"
format_stamp: "Formatted at 2026-06-20 18:30:53 on dist-test-slave-mcnm"
server_key: "cf6430bc802b5480aad4d0b15b10f88f"
server_key_iv: "9b4fbc6f64d6c99e6289fe51c34251e4"
server_key_version: "encryptionkey@0"
I20260620 18:30:53.512424 10909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:53.540643 10909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:53.545358 10909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:53.545686 10909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:53.546723 10909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:53.548286 10909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:53.548406 10909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.548579 10909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:53.548662 10909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:53.582739 10909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41653
I20260620 18:30:53.582825 11036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41653 every 8 connection(s)
I20260620 18:30:53.584060 10909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:30:53.590778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 10909
I20260620 18:30:53.590981 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:30:53.591466 388 external_mini_cluster.cc:1428] Setting key e54e1a96aa017eaa80fefa9b713ad2a5
Jun 20 18:30:53 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:53.604558 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:53.586566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60061 (local address 127.0.97.62:36503)
0620 18:30:53.586743 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:53.586747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:53.587931 (+ 1184us) server_negotiation.cc:407] Connection header received
0620 18:30:53.589899 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:53.589905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:53.589993 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:53.590098 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:53.592123 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.592920 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.594721 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.595178 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.595308 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:53.598722 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:53.598726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:53.598729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:53.598759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:53.601525 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:53.602267 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:53.602277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:53.602279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:53.602397 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:53.602808 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:53.602813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:53.602817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:53.603070 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:53.603219 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:53.603952 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 18:30:53.604168 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:30:53.605497 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:53.587041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36503 (local address 127.0.97.2:60061)
0620 18:30:53.587779 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:53.587816 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:53.589601 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:53.590319 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:53.590328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:53.591104 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:53.591903 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:53.591928 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.593111 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.593142 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:53.594571 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:53.594582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:53.595400 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:53.595408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:53.595700 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:53.596363 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:53.596384 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:53.598482 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:53.601670 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:53.601679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:53.601696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:53.602137 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:53.602503 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:53.602509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:53.602512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:53.602693 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:53.603197 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:53.603205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:53.603586 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 18:30:53.603848 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:30:53.604361 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":266,"threads_started":1}
I20260620 18:30:53.606957 11037 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36503
I20260620 18:30:53.607275 11037 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:53.608132 11037 heartbeater.cc:511] Master 127.0.97.62:36503 requested a full tablet report, sending...
I20260620 18:30:53.609774 10714 ts_manager.cc:194] Registered new tserver with Master: d36511a534054e19a7777f1eeabd4da9 (127.0.97.2:41653)
I20260620 18:30:53.610721 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60061
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:30:53.643898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:30:53.967450 11042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:30:53.967753 11042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:30:53.967926 11042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:30:53.975106 11042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:30:53.975208 11042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:30:53.975437 11042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:30:53.985271 11042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:30:53.986555 11042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:30:53.987955 11042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:30:54.001295 11048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:54.001470 11050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:30:54.001633 11047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:30:54.002540 11042 server_base.cc:1034] running on GCE node
I20260620 18:30:54.003247 11042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:30:54.004237 11042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:30:54.005496 11042 hybrid_clock.cc:648] HybridClock initialized: now 1781980254005423 us; error 57 us; skew 500 ppm
I20260620 18:30:54.005870 11042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:30:54 dist-test-slave-mcnm krb5kdc[10669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:30:54.009248 11042 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:30:54.010756 11042 webserver.cc:460] Webserver started at http://127.0.97.3:36619/ using document root <none> and password file <none>
I20260620 18:30:54.011456 11042 fs_manager.cc:359] Metadata directory not provided
I20260620 18:30:54.011636 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:30:54.011946 11042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:30:54.014834 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "605ccbe0b122415ea585d4038f3e2ea9"
format_stamp: "Formatted at 2026-06-20 18:30:54 on dist-test-slave-mcnm"
server_key: "4b263dbe256a41ae42341e6e1b027994"
server_key_iv: "deaeb01c1a6d85a86a230a2fb9398cf7"
server_key_version: "encryptionkey@0"
I20260620 18:30:54.015692 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "605ccbe0b122415ea585d4038f3e2ea9"
format_stamp: "Formatted at 2026-06-20 18:30:54 on dist-test-slave-mcnm"
server_key: "4b263dbe256a41ae42341e6e1b027994"
server_key_iv: "deaeb01c1a6d85a86a230a2fb9398cf7"
server_key_version: "encryptionkey@0"
I20260620 18:30:54.021093 11042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:30:54.024544 11057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:54.025849 11042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:30:54.026070 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "605ccbe0b122415ea585d4038f3e2ea9"
format_stamp: "Formatted at 2026-06-20 18:30:54 on dist-test-slave-mcnm"
server_key: "4b263dbe256a41ae42341e6e1b027994"
server_key_iv: "deaeb01c1a6d85a86a230a2fb9398cf7"
server_key_version: "encryptionkey@0"
I20260620 18:30:54.026269 11042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:30:54.040813 11042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:30:54.044684 11042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:30:54.044982 11042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:30:54.045882 11042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:30:54.047173 11042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:30:54.047274 11042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:54.047430 11042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:30:54.047508 11042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:30:54.080610 11042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45239
I20260620 18:30:54.080647 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45239 every 8 connection(s)
I20260620 18:30:54.081916 11042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:30:54.084713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 11042
I20260620 18:30:54.084911 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:30:54.085330 388 external_mini_cluster.cc:1428] Setting key 610c17940f406b84681e3444312853be
Jun 20 18:30:54 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:30:54.101840 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.084599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46069 (local address 127.0.97.62:36503)
0620 18:30:54.084813 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.084819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.086094 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 18:30:54.087942 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.087948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.088041 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.088147 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:54.089594 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.090319 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.091930 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.092311 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.092445 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.096058 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:54.096064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:54.096068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:54.096109 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:54.098597 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.099319 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.099326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.099332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.099476 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.100013 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.100019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.100023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.100327 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:54.100503 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.101200 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.101426 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:30:54.102712 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.085116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36503 (local address 127.0.97.3:46069)
0620 18:30:54.085947 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.085992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.087690 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.088390 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.088397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.088895 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:30:54.089426 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.089442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.090488 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.090492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.091754 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.091765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.092534 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.092543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.092764 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.093408 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:30:54.093433 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:30:54.095840 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:30:54.098751 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:54.098760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:54.098783 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:54.099182 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:54.099675 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:30:54.099681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:30:54.099685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:30:54.099885 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:30:54.100476 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:30:54.100484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:30:54.100810 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.101076 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.101597 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":392,"threads_started":1}
I20260620 18:30:54.104095 11170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36503
I20260620 18:30:54.104425 11170 heartbeater.cc:464] Registering TS with master...
I20260620 18:30:54.105126 11170 heartbeater.cc:511] Master 127.0.97.62:36503 requested a full tablet report, sending...
I20260620 18:30:54.106473 10714 ts_manager.cc:194] Registered new tserver with Master: 605ccbe0b122415ea585d4038f3e2ea9 (127.0.97.3:45239)
I20260620 18:30:54.107286 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46069
I20260620 18:30:54.110059 10903 heartbeater.cc:503] Master 127.0.97.62:36503 was elected leader, sending a full tablet report...
I20260620 18:30:54.112555 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:30:54.129175 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.117063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.0.97.62:36503)
0620 18:30:54.117273 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.117305 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.117793 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:30:54.117947 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.117951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.118027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.118119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:54.119600 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.120373 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.121637 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.122038 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.122138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.124303 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:54.124308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:54.124311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:54.124338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:54.126880 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.127452 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.127456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.127458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.127618 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.127937 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.127940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.127942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.128216 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:54.128380 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.128666 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.128815 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:30:54.133378 10714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:30:54.140012 10714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:30:54.163121 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.154822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.97.2:41653)
0620 18:30:54.155431 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.155442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.156004 (+ 562us) server_negotiation.cc:407] Connection header received
0620 18:30:54.156503 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.156512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.156732 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.156887 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:54.157615 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158675 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.161014 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.161957 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.162110 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.162314 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.162590 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.162764 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":367,"threads_started":1}
I20260620 18:30:54.164103 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.154609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41653 (local address 127.0.0.1:52110)
0620 18:30:54.155902 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.155920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.156295 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.157101 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.157105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.157136 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:54.157439 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.157446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158846 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.158851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.160866 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.160878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.162152 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.162157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.162337 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.162386 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.163451 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.163684 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":21,"spinlock_wait_cycles":373504,"thread_start_us":296,"threads_started":1}
I20260620 18:30:54.164535 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.151847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.97.1:41771)
0620 18:30:54.154261 (+ 2414us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.154268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.154297 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:30:54.154400 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.154411 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.154665 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.154880 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:54.156917 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158629 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.161997 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.163460 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.163628 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.163848 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.164079 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.164223 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230,"thread_start_us":2035,"threads_started":1}
I20260620 18:30:54.165045 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.151779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45239 (local address 127.0.0.1:34938)
0620 18:30:54.154448 (+ 2669us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.154460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.154592 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.156020 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.156025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.156314 (+ 289us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:54.156883 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.156896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158806 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.158810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.161807 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.161826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.164225 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.164234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.164407 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.164443 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.164535 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.164668 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2573,"mutex_wait_us":301,"spinlock_wait_cycles":569600,"thread_start_us":657,"threads_started":1}
I20260620 18:30:54.165277 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.152469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.97.3:45239)
0620 18:30:54.153699 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.153712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.154716 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 18:30:54.154810 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.154819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.155278 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.155569 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:54.157065 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158676 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.161968 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.163739 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.163861 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.164077 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.164771 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.164931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":824,"threads_started":1}
I20260620 18:30:54.165472 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.151580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41771 (local address 127.0.0.1:51782)
0620 18:30:54.153938 (+ 2358us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.153983 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.154144 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.154876 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.154885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.156264 (+ 1379us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:54.156737 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.156749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.158804 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.158810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.161806 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.161820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.163684 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.163690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.163861 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.163905 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.164019 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.164156 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"mutex_wait_us":236,"spinlock_wait_cycles":1792,"thread_start_us":2638,"threads_started":1}
I20260620 18:30:54.168478 10972 tablet_service.cc:1467] Processing CreateTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 (DEFAULT_TABLE table=test-table [id=b633e9ed625441a7bc52c03b19d35931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:54.169219 10838 tablet_service.cc:1467] Processing CreateTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 (DEFAULT_TABLE table=test-table [id=b633e9ed625441a7bc52c03b19d35931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:54.169924 10972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c2d185affa450e8aed8f7d88ce0693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:54.169821 11105 tablet_service.cc:1467] Processing CreateTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 (DEFAULT_TABLE table=test-table [id=b633e9ed625441a7bc52c03b19d35931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:30:54.170352 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c2d185affa450e8aed8f7d88ce0693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:54.171059 11105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c2d185affa450e8aed8f7d88ce0693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:30:54.181932 11188 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Bootstrap starting.
I20260620 18:30:54.184922 11189 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Bootstrap starting.
I20260620 18:30:54.186059 11188 tablet_bootstrap.cc:654] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:54.187333 11188 log.cc:824] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:54.188197 11190 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Bootstrap starting.
I20260620 18:30:54.188558 11189 tablet_bootstrap.cc:654] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:54.189834 11189 log.cc:824] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:54.191438 11190 tablet_bootstrap.cc:654] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Neither blocks nor log segments found. Creating new log.
I20260620 18:30:54.191524 11188 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: No bootstrap required, opened a new log
I20260620 18:30:54.191799 11188 ts_tablet_manager.cc:1397] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 18:30:54.192420 11190 log.cc:824] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:30:54.192775 11189 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: No bootstrap required, opened a new log
I20260620 18:30:54.193487 11189 ts_tablet_manager.cc:1397] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260620 18:30:54.195271 11190 tablet_bootstrap.cc:492] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: No bootstrap required, opened a new log
I20260620 18:30:54.195621 11190 ts_tablet_manager.cc:1397] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 18:30:54.202241 11190 raft_consensus.cc:348] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.202272 11188 raft_consensus.cc:348] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.202606 11190 raft_consensus.cc:374] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:54.202709 11188 raft_consensus.cc:374] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:54.202749 11190 raft_consensus.cc:729] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605ccbe0b122415ea585d4038f3e2ea9, State: Initialized, Role: FOLLOWER
I20260620 18:30:54.202812 11188 raft_consensus.cc:729] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c6e0776a26461f81fe15752dc13e12, State: Initialized, Role: FOLLOWER
I20260620 18:30:54.203445 11190 consensus_queue.cc:260] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.203462 11188 consensus_queue.cc:260] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.204596 11189 raft_consensus.cc:348] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.205022 11189 raft_consensus.cc:374] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:30:54.205107 11170 heartbeater.cc:503] Master 127.0.97.62:36503 was elected leader, sending a full tablet report...
I20260620 18:30:54.205145 11189 raft_consensus.cc:729] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36511a534054e19a7777f1eeabd4da9, State: Initialized, Role: FOLLOWER
I20260620 18:30:54.205304 11188 ts_tablet_manager.cc:1428] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:30:54.205789 11189 consensus_queue.cc:260] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.206804 11190 ts_tablet_manager.cc:1428] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260620 18:30:54.207880 11037 heartbeater.cc:503] Master 127.0.97.62:36503 was elected leader, sending a full tablet report...
I20260620 18:30:54.208475 11189 ts_tablet_manager.cc:1428] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260620 18:30:54.330273 10904 tablet.cc:2369] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:30:54.333460 11171 tablet.cc:2369] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:30:54.335991 11038 tablet.cc:2369] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:30:54.463789 11196 raft_consensus.cc:482] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:30:54.464103 11196 raft_consensus.cc:504] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.465968 11196 leader_election.cc:288] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c6e0776a26461f81fe15752dc13e12 (127.0.97.1:41771), 605ccbe0b122415ea585d4038f3e2ea9 (127.0.97.3:45239)
I20260620 18:30:54.472759 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.466618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39529 (local address 127.0.97.3:45239)
0620 18:30:54.466846 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.466854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.467533 (+ 679us) server_negotiation.cc:407] Connection header received
0620 18:30:54.467749 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.467752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.467808 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.467899 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:54.468528 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.469225 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.471046 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.471867 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.472015 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.472122 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.472390 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.472511 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:30:54.472934 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.466477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45239 (local address 127.0.97.2:39529)
0620 18:30:54.467395 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.467410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.467596 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.467963 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.467967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.468006 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:54.468369 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.468378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.469364 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.469370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.470884 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.470896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.472075 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.472080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.472222 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.472248 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.472360 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.472494 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":7,"thread_start_us":589,"threads_started":1}
I20260620 18:30:54.473074 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.466287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38259 (local address 127.0.97.1:41771)
0620 18:30:54.466549 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.466559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.467325 (+ 766us) server_negotiation.cc:407] Connection header received
0620 18:30:54.467604 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.467609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.467670 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.467786 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:30:54.468472 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.469205 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.471166 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.472055 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.472231 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.472354 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.472662 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.472788 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:30:54.473308 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.466226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41771 (local address 127.0.97.2:38259)
0620 18:30:54.467183 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 18:30:54.467206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:30:54.467416 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:30:54.467812 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:30:54.467817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:30:54.467844 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:30:54.468300 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.468311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.469357 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.469366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.470993 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:30:54.471006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.472296 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.472304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:30:54.472491 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.472515 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:30:54.472636 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 18:30:54.472780 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":56,"thread_start_us":560,"threads_started":1}
I20260620 18:30:54.474345 11125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c2d185affa450e8aed8f7d88ce0693" candidate_uuid: "d36511a534054e19a7777f1eeabd4da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605ccbe0b122415ea585d4038f3e2ea9" is_pre_election: true
I20260620 18:30:54.474326 10858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c2d185affa450e8aed8f7d88ce0693" candidate_uuid: "d36511a534054e19a7777f1eeabd4da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c6e0776a26461f81fe15752dc13e12" is_pre_election: true
I20260620 18:30:54.474942 11125 raft_consensus.cc:2454] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36511a534054e19a7777f1eeabd4da9 in term 0.
I20260620 18:30:54.474942 10858 raft_consensus.cc:2454] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36511a534054e19a7777f1eeabd4da9 in term 0.
I20260620 18:30:54.475760 10927 leader_election.cc:302] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c6e0776a26461f81fe15752dc13e12, d36511a534054e19a7777f1eeabd4da9; no voters:
I20260620 18:30:54.476235 11196 raft_consensus.cc:2781] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:30:54.476357 11196 raft_consensus.cc:482] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:30:54.476513 11196 raft_consensus.cc:3037] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:54.478719 11196 raft_consensus.cc:504] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.479321 11196 leader_election.cc:288] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [CANDIDATE]: Term 1 election: Requested vote from peers 33c6e0776a26461f81fe15752dc13e12 (127.0.97.1:41771), 605ccbe0b122415ea585d4038f3e2ea9 (127.0.97.3:45239)
I20260620 18:30:54.479825 10858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c2d185affa450e8aed8f7d88ce0693" candidate_uuid: "d36511a534054e19a7777f1eeabd4da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c6e0776a26461f81fe15752dc13e12"
I20260620 18:30:54.479828 11125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c2d185affa450e8aed8f7d88ce0693" candidate_uuid: "d36511a534054e19a7777f1eeabd4da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605ccbe0b122415ea585d4038f3e2ea9"
I20260620 18:30:54.480038 10858 raft_consensus.cc:3037] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:54.480073 11125 raft_consensus.cc:3037] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:30:54.482203 10858 raft_consensus.cc:2454] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36511a534054e19a7777f1eeabd4da9 in term 1.
I20260620 18:30:54.482417 11125 raft_consensus.cc:2454] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36511a534054e19a7777f1eeabd4da9 in term 1.
I20260620 18:30:54.482641 10927 leader_election.cc:302] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c6e0776a26461f81fe15752dc13e12, d36511a534054e19a7777f1eeabd4da9; no voters:
I20260620 18:30:54.482924 11196 raft_consensus.cc:2781] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:30:54.483373 11196 raft_consensus.cc:686] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 1 LEADER]: Becoming Leader. State: Replica: d36511a534054e19a7777f1eeabd4da9, State: Running, Role: LEADER
I20260620 18:30:54.483834 11196 consensus_queue.cc:237] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } }
I20260620 18:30:54.488135 10713 catalog_manager.cc:5495] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d36511a534054e19a7777f1eeabd4da9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d36511a534054e19a7777f1eeabd4da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36511a534054e19a7777f1eeabd4da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41653 } health_report { overall_health: HEALTHY } } }
I20260620 18:30:54.530974 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.525489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.0.97.2:41653)
0620 18:30:54.525736 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.525743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.525869 (+ 126us) server_negotiation.cc:407] Connection header received
0620 18:30:54.526132 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.526136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.526210 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.526376 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:30:54.526740 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.527483 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.528835 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.529238 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.529391 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.529874 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:30:54.530400 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:30:54.530558 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.530612 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.530696 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:30:54.539323 11125 raft_consensus.cc:1261] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 1 FOLLOWER]: Refusing update from remote peer d36511a534054e19a7777f1eeabd4da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:54.539322 10858 raft_consensus.cc:1261] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 1 FOLLOWER]: Refusing update from remote peer d36511a534054e19a7777f1eeabd4da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:30:54.540346 11199 consensus_queue.cc:1035] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605ccbe0b122415ea585d4038f3e2ea9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:54.540694 11196 consensus_queue.cc:1035] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c6e0776a26461f81fe15752dc13e12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:30:54.565508 10713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:54.565763 10713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:30:54.569047 10713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bd1e05c00db4661b6d6ada36637fbb8: Sending DeleteTablet for 3 replicas of tablet c9c2d185affa450e8aed8f7d88ce0693
I20260620 18:30:54.570000 10838 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51782
I20260620 18:30:54.570158 11105 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34938
I20260620 18:30:54.570401 10972 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c2d185affa450e8aed8f7d88ce0693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:30:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52110
I20260620 18:30:54.572587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10775
I20260620 18:30:54.580900 11214 tablet_replica.cc:330] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: stopping tablet replica
I20260620 18:30:54.581430 11214 raft_consensus.cc:2229] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:30:54.583422 11213 tablet_replica.cc:330] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: stopping tablet replica
I20260620 18:30:54.583993 11213 raft_consensus.cc:2229] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:54.584569 11213 raft_consensus.cc:2258] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:54.584812 11212 tablet_replica.cc:330] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: stopping tablet replica
I20260620 18:30:54.585216 11212 raft_consensus.cc:2229] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:30:54.585729 11212 raft_consensus.cc:2258] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:54.586416 11214 raft_consensus.cc:2258] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:30:54.587320 11213 ts_tablet_manager.cc:1905] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:54.588332 11214 ts_tablet_manager.cc:1905] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:54.588429 11212 ts_tablet_manager.cc:1905] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:30:54.594475 11213 ts_tablet_manager.cc:1918] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:54.594681 11213 log.cc:1196] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/c9c2d185affa450e8aed8f7d88ce0693
I20260620 18:30:54.595273 11213 ts_tablet_manager.cc:1939] T c9c2d185affa450e8aed8f7d88ce0693 P 605ccbe0b122415ea585d4038f3e2ea9: Deleting consensus metadata
I20260620 18:30:54.597468 10699 catalog_manager.cc:4841] TS 605ccbe0b122415ea585d4038f3e2ea9 (127.0.97.3:45239): tablet c9c2d185affa450e8aed8f7d88ce0693 (table test-table [id=b633e9ed625441a7bc52c03b19d35931]) successfully deleted
I20260620 18:30:54.599251 11212 ts_tablet_manager.cc:1918] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:54.599413 11212 log.cc:1196] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/c9c2d185affa450e8aed8f7d88ce0693
I20260620 18:30:54.600013 11212 ts_tablet_manager.cc:1939] T c9c2d185affa450e8aed8f7d88ce0693 P 33c6e0776a26461f81fe15752dc13e12: Deleting consensus metadata
I20260620 18:30:54.601773 10700 catalog_manager.cc:4841] TS 33c6e0776a26461f81fe15752dc13e12 (127.0.97.1:41771): tablet c9c2d185affa450e8aed8f7d88ce0693 (table test-table [id=b633e9ed625441a7bc52c03b19d35931]) successfully deleted
Jun 20 18:30:54 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:30:54.609162 11214 ts_tablet_manager.cc:1918] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:30:54.609431 11214 log.cc:1196] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/c9c2d185affa450e8aed8f7d88ce0693
I20260620 18:30:54.610059 11214 ts_tablet_manager.cc:1939] T c9c2d185affa450e8aed8f7d88ce0693 P d36511a534054e19a7777f1eeabd4da9: Deleting consensus metadata
I20260620 18:30:54.611289 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.574238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.97.1:41771)
0620 18:30:54.574452 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.574460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.585309 (+ 10849us) server_negotiation.cc:407] Connection header received
0620 18:30:54.585598 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.585604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.585694 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.585764 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:54.589024 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.590165 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.593274 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.593954 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.594098 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.603815 (+ 9717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:54.603829 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:54.603846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:54.603886 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:54.607764 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.608639 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.608648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.608651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.608813 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.609316 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.609321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.609324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.609803 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:54.610102 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.610701 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.610919 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:30:54.611977 10700 catalog_manager.cc:4841] TS d36511a534054e19a7777f1eeabd4da9 (127.0.97.2:41653): tablet c9c2d185affa450e8aed8f7d88ce0693 (table test-table [id=b633e9ed625441a7bc52c03b19d35931]) successfully deleted
I20260620 18:30:54.612697 10898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:54.701871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10775
I20260620 18:30:54.725621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10909
Jun 20 18:30:54 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:30:54.743286 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.726445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.0.97.2:41653)
0620 18:30:54.726621 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.726625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.726648 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:30:54.727030 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.727035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.727110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.727212 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:54.728847 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.729874 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.731843 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.732475 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.732581 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.737388 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:54.737404 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:54.737420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:54.737464 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:54.740555 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.741226 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.741232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.741235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.741363 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.741772 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.741778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.741782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.742121 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:54.742476 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.742711 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.742937 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:30:54.743960 11032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:54.859865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10909
I20260620 18:30:54.880956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 11042
Jun 20 18:30:54 dist-test-slave-mcnm krb5kdc[10669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:30:54.896324 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:30:54.881727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.0.97.3:45239)
0620 18:30:54.881941 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:30:54.881946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:30:54.881982 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:30:54.882145 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:30:54.882149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:30:54.882219 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:30:54.882302 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:30:54.883459 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.884219 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.885514 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:30:54.885882 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:30:54.885984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:30:54.890519 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:30:54.890534 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:30:54.890550 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:30:54.890583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:30:54.893350 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.894079 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.894087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.894089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.894258 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:30:54.894799 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:30:54.894803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:30:54.894806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:30:54.895174 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:30:54.895436 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 18:30:54.895891 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:30:54.896056 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:30:54.896974 11165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:54.984427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 11042
I20260620 18:30:55.001494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 10682
I20260620 18:30:55.002607 10744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:30:55.112432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 10682
2026-06-20T18:30:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4441 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:30:55 dist-test-slave-mcnm krb5kdc[11222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:30:55 dist-test-slave-mcnm krb5kdc[11222](info): set up 2 sockets
Jun 20 18:30:55 dist-test-slave-mcnm krb5kdc[11222](info): commencing operation
krb5kdc: starting...
W20260620 18:30:56.216351 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:30:56 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:30:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 18:30:56.535466 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/postgres -l logfile start
2026-06-20 18:30:59.099 UTC [11252] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:30:59.099 UTC [11252] LOG: listening on IPv4 address "127.0.97.20", port 33053
2026-06-20 18:30:59.103 UTC [11252] LOG: listening on Unix socket "/tmp/.s.PGSQL.33053"
2026-06-20 18:30:59.127 UTC [11255] LOG: database system was shut down at 2026-06-20 18:30:57 UTC
2026-06-20 18:30:59.130 UTC [11252] LOG: database system is ready to accept connections
I20260620 18:31:00.104022 388 mini_postgres.cc:96] Postgres bound to 33053
2026-06-20 18:31:00.121 UTC [11263] FATAL: database "slave" does not exist
127.0.97.20:33053 - accepting connections
I20260620 18:31:00.121909 388 mini_ranger.cc:161] Starting Ranger...
I20260620 18:31:00.147421 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:31:00.643950 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:31:00.644089 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin
2026-06-20 18:31:00,926 [I] DB FLAVOR :POSTGRES
2026-06-20 18:31:00,926 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:31:00,926 [I] Checking connection..
2026-06-20 18:31:00,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:31:01,193 [I] Checking connection passed.
2026-06-20 18:31:01,193 [I] --------- Verifying version history table ---------
2026-06-20 18:31:01,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:01,448 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:31:01,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:01,706 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:31:01,707 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:31:01,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:31:01.959 UTC [11327] WARNING: there is no transaction in progress
2026-06-20 18:31:01,965 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:31:01,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:02,244 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:31:02,244 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:31:02,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:31:02,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:31:02,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mcnm.c.gcp-upstream.internal','N') ;"
2026-06-20 18:31:02,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:31:03,263 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:31:03,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:31:03,504 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:31:03,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:31:03,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:31:03,991 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:31:03,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:31:04.917 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:04.935 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:04.952 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:04.968 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:04.984 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.004 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.059 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.066 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.080 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.089 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.099 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.113 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.123 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.129 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.136 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05.142 UTC [11435] WARNING: there is no transaction in progress
2026-06-20 18:31:05,806 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:31:05,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:31:06,048 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:31:06,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:31:06,296 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:31:06,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:31:06,541 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:31:06,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:31:06,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:31:07,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mcnm.c.gcp-upstream.internal';"
2026-06-20 18:31:07,262 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:31:07,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:31:07,506 [I] DB_PATCHES have already been applied
I20260620 18:31:07.512506 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcfg4Mf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:31:07.512591 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 18:31:07.528345 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:56925
Jun 20, 2026 6:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:31:08.670362 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.549s user 0.002s sys 0.036s
Jun 20, 2026 6:31:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:31:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:31:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:31:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:31:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:31:29.600730 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:31:29 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980289, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:31:29.790062 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:31:29.810825 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:31:30.680913 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:31:30.681087 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms
I20260620 18:31:30.681241 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33053
2026-06-20 18:31:31,102 [I] DB FLAVOR :POSTGRES
2026-06-20 18:31:31,102 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:31:31,102 [I] Checking connection
2026-06-20 18:31:31,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:31:31,350 [I] connection success
2026-06-20 18:31:31,351 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:31:31,351 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:31:31,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:31:31,606 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:31:31,607 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:31:31,607 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:31:31,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33053/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:31:31,888 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:31:32.396543 388 mini_ranger_kms.cc:310] Created kms service
I20260620 18:31:32.606107 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:31:32.651077 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:31:32 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980289, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:33.032270 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:31:33.032615 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:31:33.032697 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 18:31:33.052901 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11615
I20260620 18:31:33.053079 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:42009
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:31:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:31:35.372225 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.582s user 0.128s sys 0.310s
I20260620 18:31:35.372473 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:31:35.372684 388 mini_ranger_kms.cc:401] 127.0.97.20:42009/kms/v1/keys
Jun 20 18:31:35 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980295, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:31:35 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980295, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:31:36 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980295, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:31:38 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:31:38.504040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43807
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43807
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42009/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:38.870639 11666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:38.870954 11666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:38.871096 11666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:38.878489 11666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:31:38.878613 11666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:38.878687 11666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:38.878803 11666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:31:38.878868 11666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:31:38.889650 11666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:42009/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43807
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:31:38.891079 11666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:38.892570 11666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:38.905279 11671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:38.905243 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:38.905507 11672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:38.906340 11666 server_base.cc:1034] running on GCE node
I20260620 18:31:38.907279 11666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:38.909055 11666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:38.910336 11666 hybrid_clock.cc:648] HybridClock initialized: now 1781980298910293 us; error 69 us; skew 500 ppm
I20260620 18:31:38.910781 11666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:38 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:38.919064 11666 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:31:38.920835 11666 webserver.cc:460] Webserver started at http://127.0.97.62:39683/ using document root <none> and password file <none>
I20260620 18:31:38.921547 11666 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:38.922079 11666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:38.922413 11666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:31:38 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:38.979214 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "9cc711ab7639491db64f82df27821c9e"
format_stamp: "Formatted at 2026-06-20 18:31:38 on dist-test-slave-mcnm"
server_key: "b2facd6c28f16f4db45b213183168b87"
server_key_iv: "d00506d1bde99d17927db632444f6a98"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:38.980160 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cc711ab7639491db64f82df27821c9e"
format_stamp: "Formatted at 2026-06-20 18:31:38 on dist-test-slave-mcnm"
server_key: "b2facd6c28f16f4db45b213183168b87"
server_key_iv: "d00506d1bde99d17927db632444f6a98"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:38.985759 11666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:31:39.006961 11681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:39.008827 11666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:31:39.009050 11666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "9cc711ab7639491db64f82df27821c9e"
format_stamp: "Formatted at 2026-06-20 18:31:38 on dist-test-slave-mcnm"
server_key: "b2facd6c28f16f4db45b213183168b87"
server_key_iv: "d00506d1bde99d17927db632444f6a98"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:39.009280 11666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:39.023396 11666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:39.027937 11666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:39.028250 11666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:39.048637 11666 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43807
I20260620 18:31:39.048676 11732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43807 every 8 connection(s)
I20260620 18:31:39.050001 11666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:31:39.053805 11733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:39.056010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 11666
I20260620 18:31:39.056212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:31:39.056999 388 external_mini_cluster.cc:1428] Setting key 98d0e74602db45679e710b1ba93ca1ad
I20260620 18:31:39.066529 11733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: Bootstrap starting.
I20260620 18:31:39.069839 11733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:39.070955 11733 log.cc:824] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:31:39 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:31:39.074033 11733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: No bootstrap required, opened a new log
I20260620 18:31:39.079632 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:39.059229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.97.62:43807)
0620 18:31:39.060113 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:39.060126 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:39.060170 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:31:39.061871 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:39.061894 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:39.062290 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:39.062696 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:39.064115 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:39.065886 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:39.067243 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:39.067987 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:39.068187 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:39.072372 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:39.072380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:39.072394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:39.072428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:39.075341 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:39.076220 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:39.076226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:39.076232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:39.076391 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:39.076790 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:39.076793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:39.076795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:39.077323 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:39.077586 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:39.078038 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 18:31:39.078402 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":429,"threads_started":1}
I20260620 18:31:39.083140 11733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260620 18:31:39.083552 11733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:39.083681 11733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc711ab7639491db64f82df27821c9e, State: Initialized, Role: FOLLOWER
I20260620 18:31:39.084255 11733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260620 18:31:39.084457 11733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:31:39.084584 11733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:31:39.084769 11733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:39.086846 11733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260620 18:31:39.087348 11733 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc711ab7639491db64f82df27821c9e; no voters:
I20260620 18:31:39.087967 11733 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:31:39.088068 11738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:31:39.088759 11738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [term 1 LEADER]: Becoming Leader. State: Replica: 9cc711ab7639491db64f82df27821c9e, State: Running, Role: LEADER
I20260620 18:31:39.089279 11738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260620 18:31:39.090049 11733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:31:39.092607 11740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc711ab7639491db64f82df27821c9e. Latest consensus state: current_term: 1 leader_uuid: "9cc711ab7639491db64f82df27821c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } } }
I20260620 18:31:39.092687 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc711ab7639491db64f82df27821c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc711ab7639491db64f82df27821c9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } } }
I20260620 18:31:39.093024 11740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:39.093062 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:39.100605 11744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:31:39.105335 11744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:31:39.106096 11733 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:31:39.124645 11744 catalog_manager.cc:1347] Generated new cluster ID: 92c00302bd92409ea2451ef5c417ae47
I20260620 18:31:39.124922 11744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:31:39.154983 11744 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:31:39.157043 11744 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:31:39.166568 11744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: Generated new TSK 0
I20260620 18:31:39.167786 11744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:31:39.933555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:31:40 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:31:40.393833 11778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:40.394328 11778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:40.394506 11778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:40.408178 11778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:40.408349 11778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:40.408612 11778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:31:40.426884 11778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:42009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:31:40.428728 11778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:40.430574 11778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:40.463042 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:40.467921 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:40.468771 11778 server_base.cc:1034] running on GCE node
W20260620 18:31:40.470913 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:40.471834 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:40.474982 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:40.476269 11778 hybrid_clock.cc:648] HybridClock initialized: now 1781980300476095 us; error 159 us; skew 500 ppm
I20260620 18:31:40.476730 11778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:40 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:40.480870 11778 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:31:40.484328 11778 webserver.cc:460] Webserver started at http://127.0.97.1:33719/ using document root <none> and password file <none>
I20260620 18:31:40.485270 11778 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:40.485417 11778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:40.486195 11778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:31:40 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:40.504241 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "117c3434df4c4817b7ee1638641ebe1a"
format_stamp: "Formatted at 2026-06-20 18:31:40 on dist-test-slave-mcnm"
server_key: "076b64c34be888769fc9a6df029790e3"
server_key_iv: "1a186281ca06bff771d0fcfb35f8e703"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:40.505132 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "117c3434df4c4817b7ee1638641ebe1a"
format_stamp: "Formatted at 2026-06-20 18:31:40 on dist-test-slave-mcnm"
server_key: "076b64c34be888769fc9a6df029790e3"
server_key_iv: "1a186281ca06bff771d0fcfb35f8e703"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:40.513306 11778 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 18:31:40.535096 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:40.544895 11778 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260620 18:31:40.545251 11778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "117c3434df4c4817b7ee1638641ebe1a"
format_stamp: "Formatted at 2026-06-20 18:31:40 on dist-test-slave-mcnm"
server_key: "076b64c34be888769fc9a6df029790e3"
server_key_iv: "1a186281ca06bff771d0fcfb35f8e703"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:40.545629 11778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:40.594359 11778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:40.600095 11778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:40.600462 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:40.601692 11778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:40.603503 11778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:40.603634 11778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:31:40.603809 11778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:40.603883 11778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:31:40 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:40.795956 11778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39601
I20260620 18:31:40.795957 11910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39601 every 8 connection(s)
I20260620 18:31:40.797345 11778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:31:40.804261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 11778
I20260620 18:31:40.804522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:31:40.805511 388 external_mini_cluster.cc:1428] Setting key 2d414ee961c2a25cb5e38cf528bdbac9
Jun 20 18:31:40 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:31:40.842312 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:40.802280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47439 (local address 127.0.97.62:43807)
0620 18:31:40.817882 (+ 15602us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:40.817893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:40.820758 (+ 2865us) server_negotiation.cc:407] Connection header received
0620 18:31:40.823819 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:40.823824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:40.823911 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:40.823994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:40.825912 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:40.826663 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:40.828546 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:40.829224 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:40.829367 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:40.834230 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:40.834236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:40.834241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:40.834294 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:40.837901 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:40.839050 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:40.839056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:40.839060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:40.839226 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:40.839848 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:40.839854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:40.839858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:40.840201 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:40.840356 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:40.841223 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 18:31:40.841481 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15409,"thread_start_us":15229,"threads_started":1}
I20260620 18:31:40.842651 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:40.801515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.1:47439)
0620 18:31:40.818700 (+ 17185us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:40.818774 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:40.822599 (+ 3825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:40.824602 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:40.824611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:40.825092 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:40.825719 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:40.825738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:40.826864 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:40.826868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:40.828279 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:40.828293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:40.829516 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:40.829523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:40.829739 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:40.830684 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:40.830718 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:40.833934 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:40.838097 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:40.838110 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:40.838134 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:40.838879 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:40.839376 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:40.839381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:40.839385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:40.839696 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:40.840366 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:40.840373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:40.840804 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 18:31:40.841076 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:31:40.841499 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16721,"thread_start_us":21733,"threads_started":1}
I20260620 18:31:40.844987 11911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260620 18:31:40.845367 11911 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:40.846294 11911 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:31:40.849030 11698 ts_manager.cc:194] Registered new tserver with Master: 117c3434df4c4817b7ee1638641ebe1a (127.0.97.1:39601)
I20260620 18:31:40.850726 11698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:31:40.883093 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 18:31:41.341214 11917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:41.341496 11917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:41.341651 11917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:41.349176 11917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:41.349283 11917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:41.349447 11917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:31:41.360631 11917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:42009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:31:41.361898 11917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:41.363176 11917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:41.376786 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:41.377035 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:41.376833 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:41.377792 11917 server_base.cc:1034] running on GCE node
I20260620 18:31:41.378425 11917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:41.379465 11917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:41.380718 11917 hybrid_clock.cc:648] HybridClock initialized: now 1781980301380652 us; error 60 us; skew 500 ppm
I20260620 18:31:41.381062 11917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:41.384271 11917 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:31:41.385885 11917 webserver.cc:460] Webserver started at http://127.0.97.2:34089/ using document root <none> and password file <none>
I20260620 18:31:41.386531 11917 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:41.386633 11917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:41.386893 11917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:41.405355 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "05f8054001974254916b4912753fe5f0"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "455dcb4d75f67737933f1b3a7d317a75"
server_key_iv: "2a4d5e1410149e9afbeffd3c5360a7b3"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:41.406500 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "05f8054001974254916b4912753fe5f0"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "455dcb4d75f67737933f1b3a7d317a75"
server_key_iv: "2a4d5e1410149e9afbeffd3c5360a7b3"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:41.413990 11917 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 18:31:41.430099 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:41.432426 11917 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 18:31:41.432655 11917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "05f8054001974254916b4912753fe5f0"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "455dcb4d75f67737933f1b3a7d317a75"
server_key_iv: "2a4d5e1410149e9afbeffd3c5360a7b3"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:41.432936 11917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:41.458420 11917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:41.462530 11917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:41.462847 11917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:41.463891 11917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:41.465281 11917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:41.465350 11917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:41.465507 11917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:41.465556 11917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:41.503515 11917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40949
I20260620 18:31:41.503577 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40949 every 8 connection(s)
I20260620 18:31:41.505306 11917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:31:41.507158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 11917
I20260620 18:31:41.507349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:31:41.507881 388 external_mini_cluster.cc:1428] Setting key 6f77e1675fdc5d1db9153110571b505f
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:31:41.531879 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:41.508227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33293 (local address 127.0.97.62:43807)
0620 18:31:41.514723 (+ 6496us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:41.514729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:41.514746 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:31:41.514812 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:41.514817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:41.514887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:41.515011 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:41.517548 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:41.518774 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:41.520991 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:41.521395 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:41.521571 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:41.525587 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:41.525592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:41.525595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:41.525627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:41.528310 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:41.529071 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:41.529076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:41.529081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:41.529208 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:41.530051 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:41.530055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:41.530057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:41.530336 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:41.530497 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:41.531237 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 18:31:41.531460 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6380,"thread_start_us":6267,"threads_started":1}
I20260620 18:31:41.533128 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:41.508995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.2:33293)
0620 18:31:41.511565 (+ 2570us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:41.511614 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:41.514022 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:41.515280 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:41.515736 (+ 456us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:41.516407 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:41.517283 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:41.517314 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:41.518963 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:41.518971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:41.520828 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:41.520848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:41.522197 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:41.522202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:41.522417 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:41.523125 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:41.523152 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:41.525332 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:41.528468 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:41.528477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:41.528495 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:41.528930 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:41.529492 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:41.529496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:41.529499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:41.529770 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:41.530473 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:41.530481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:41.530845 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 18:31:41.531149 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 18:31:41.531748 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":1105,"threads_started":1}
I20260620 18:31:41.534669 12054 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260620 18:31:41.535020 12054 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:41.535837 12054 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:31:41.537772 11698 ts_manager.cc:194] Registered new tserver with Master: 05f8054001974254916b4912753fe5f0 (127.0.97.2:40949)
I20260620 18:31:41.538713 11698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:31:41.571975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:31:41.858825 11911 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
W20260620 18:31:41.927403 12060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:41.927753 12060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:41.927848 12060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:41.935480 12060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:41.935716 12060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:41.935935 12060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:31:41.946542 12060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:42009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:31:41.947875 12060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:41.949110 12060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:41.962710 12066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:41.962849 12065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:41.962795 12068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:41.963833 12060 server_base.cc:1034] running on GCE node
I20260620 18:31:41.964468 12060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:41.965356 12060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:41.966611 12060 hybrid_clock.cc:648] HybridClock initialized: now 1781980301966570 us; error 35 us; skew 500 ppm
I20260620 18:31:41.966962 12060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:41.970206 12060 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:31:41.971797 12060 webserver.cc:460] Webserver started at http://127.0.97.3:46091/ using document root <none> and password file <none>
I20260620 18:31:41.972471 12060 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:41.972622 12060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:41.973040 12060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:31:41 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:31:41.988786 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "1a164ee49ea34cb19d456dc5d789aa9c"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "f030c743adb1ec1ec0bbd2f95db6f6b0"
server_key_iv: "829c3a25ad5dd7046b4c558e90dff5d7"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:41.989750 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1a164ee49ea34cb19d456dc5d789aa9c"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "f030c743adb1ec1ec0bbd2f95db6f6b0"
server_key_iv: "829c3a25ad5dd7046b4c558e90dff5d7"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:41.995503 12060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:31:42.012372 12075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:42.014266 12060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:31:42.014453 12060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "1a164ee49ea34cb19d456dc5d789aa9c"
format_stamp: "Formatted at 2026-06-20 18:31:41 on dist-test-slave-mcnm"
server_key: "f030c743adb1ec1ec0bbd2f95db6f6b0"
server_key_iv: "829c3a25ad5dd7046b4c558e90dff5d7"
server_key_version: "kuduclusterkey@0"
I20260620 18:31:42.014705 12060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:42.038292 12060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:42.042559 12060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:42.042894 12060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:42.043834 12060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:42.045121 12060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:42.045192 12060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:42.045362 12060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:42.045410 12060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:42.080613 12060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46187
I20260620 18:31:42.080705 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46187 every 8 connection(s)
I20260620 18:31:42.081939 12060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:31:42.088544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 12060
I20260620 18:31:42.088725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:31:42.089272 388 external_mini_cluster.cc:1428] Setting key da1aed69879bc634ea91f8d3779cdc9a
Jun 20 18:31:42 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:31:42.102869 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.084413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33377 (local address 127.0.97.62:43807)
0620 18:31:42.084877 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.084882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.085838 (+ 956us) server_negotiation.cc:407] Connection header received
0620 18:31:42.087683 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.087687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.087764 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.087863 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:42.089726 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.090701 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.092960 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.093394 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.093500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.096868 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:42.096873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:42.096877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:42.096909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:42.100190 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.100928 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.100931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.100933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.101062 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.101446 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.101449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.101451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.101708 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:42.101817 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.102410 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.102564 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":204,"threads_started":1}
I20260620 18:31:42.104048 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.084981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.3:33377)
0620 18:31:42.085670 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.085710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.087365 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.088087 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.088095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.088719 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:42.089546 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.089571 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.090869 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.090875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.092803 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.092814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.093594 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.093599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.093803 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.094514 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:42.094538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:42.096671 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:42.100305 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:42.100313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:42.100326 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:42.100819 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:42.101154 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:42.101156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:42.101159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:42.101343 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:42.101812 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:42.101818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:42.102110 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.102322 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.102687 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260620 18:31:42.105605 12188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260620 18:31:42.105906 12188 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:42.106693 12188 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
I20260620 18:31:42.108065 11698 ts_manager.cc:194] Registered new tserver with Master: 1a164ee49ea34cb19d456dc5d789aa9c (127.0.97.3:46187)
I20260620 18:31:42.108893 11698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33377
I20260620 18:31:42.117475 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:31:42.135627 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.122111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.0.97.62:43807)
0620 18:31:42.122344 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.122350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.123102 (+ 752us) server_negotiation.cc:407] Connection header received
0620 18:31:42.123290 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.123297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.123391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.123518 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:42.125124 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.125836 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.127243 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.127730 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.127827 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.130189 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:42.130194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:42.130197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:42.130228 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:42.133246 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.133930 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.133934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.133936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.134048 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.134391 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.134394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.134396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.134642 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:42.134817 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.135179 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.135333 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:31:42.140198 11698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:31:42.147079 11698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:31:42.172474 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.160160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46187 (local address 127.0.0.1:50830)
0620 18:31:42.161690 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.161748 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.161955 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.163965 (+ 2010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.163973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.164004 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:31:42.164360 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.164370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.166420 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.166425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.169127 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.169138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.171403 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.171411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.171650 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.171719 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.171847 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.172065 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":74,"thread_start_us":3112,"threads_started":1}
I20260620 18:31:42.173722 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.160774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39601 (local address 127.0.0.1:55160)
0620 18:31:42.163573 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.163592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.163705 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.164515 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.164522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.164550 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:31:42.164920 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.164931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.166806 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.166812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.169084 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.169102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.172905 (+ 3803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.172915 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.173084 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.173113 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.173201 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.173328 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"mutex_wait_us":63,"thread_start_us":949,"threads_started":1}
I20260620 18:31:42.173790 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.160184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.0.97.3:46187)
0620 18:31:42.163247 (+ 3063us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.163258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.163298 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:31:42.163400 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.163407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.163725 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.163945 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:31:42.164594 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.166265 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.169275 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.171175 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.172979 (+ 1804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.173210 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.173278 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.173448 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":2754,"threads_started":1}
I20260620 18:31:42.174624 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.161518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.0.97.1:39601)
0620 18:31:42.163868 (+ 2350us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.163876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.163904 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:31:42.164003 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.164011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.164279 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.164520 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:31:42.165114 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.166644 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.169275 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.172087 (+ 2812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.172213 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.172477 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.174141 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.174332 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":912,"threads_started":1}
I20260620 18:31:42.178891 12123 tablet_service.cc:1467] Processing CreateTablet for tablet b719e893697542d6ac9160d718394e95 (DEFAULT_TABLE table=test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:31:42.178893 11846 tablet_service.cc:1467] Processing CreateTablet for tablet b719e893697542d6ac9160d718394e95 (DEFAULT_TABLE table=test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:31:42.180289 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b719e893697542d6ac9160d718394e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:42.180320 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b719e893697542d6ac9160d718394e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:42.182540 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.163906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.0.97.2:40949)
0620 18:31:42.175320 (+ 11414us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.175328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.175356 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:31:42.175474 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.175483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.175765 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.176077 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:31:42.176396 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.177928 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.180057 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.181448 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.181590 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.181774 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.182051 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.182206 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11268,"thread_start_us":11166,"threads_started":1}
I20260620 18:31:42.182363 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.164948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40949 (local address 127.0.0.1:55500)
0620 18:31:42.172693 (+ 7745us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.172711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.172798 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.176080 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.176086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.176112 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:31:42.176271 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.176281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.178095 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.178102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.179867 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.179884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.181637 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.181643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.181824 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.181867 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.181974 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.182112 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7652,"thread_start_us":10591,"threads_started":1}
I20260620 18:31:42.187310 11989 tablet_service.cc:1467] Processing CreateTablet for tablet b719e893697542d6ac9160d718394e95 (DEFAULT_TABLE table=test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:31:42.188720 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b719e893697542d6ac9160d718394e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:42.192304 12207 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Bootstrap starting.
I20260620 18:31:42.194666 12208 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Bootstrap starting.
I20260620 18:31:42.196592 12207 tablet_bootstrap.cc:654] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:42.197858 12207 log.cc:824] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:31:42.197965 12208 tablet_bootstrap.cc:654] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:42.199167 12208 log.cc:824] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:31:42.201722 12207 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: No bootstrap required, opened a new log
I20260620 18:31:42.202004 12207 ts_tablet_manager.cc:1397] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:31:42.203936 12208 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: No bootstrap required, opened a new log
I20260620 18:31:42.204344 12208 ts_tablet_manager.cc:1397] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 18:31:42.210420 12211 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Bootstrap starting.
I20260620 18:31:42.214834 12211 tablet_bootstrap.cc:654] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:42.214787 12207 raft_consensus.cc:348] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.215224 12207 raft_consensus.cc:374] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:42.215333 12207 raft_consensus.cc:729] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c3434df4c4817b7ee1638641ebe1a, State: Initialized, Role: FOLLOWER
I20260620 18:31:42.215595 12208 raft_consensus.cc:348] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.216094 12208 raft_consensus.cc:374] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:42.216187 12208 raft_consensus.cc:729] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a164ee49ea34cb19d456dc5d789aa9c, State: Initialized, Role: FOLLOWER
I20260620 18:31:42.216001 12207 consensus_queue.cc:260] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.216753 12211 log.cc:824] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:31:42.216894 12208 consensus_queue.cc:260] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.221010 12211 tablet_bootstrap.cc:492] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: No bootstrap required, opened a new log
I20260620 18:31:42.222929 12208 ts_tablet_manager.cc:1428] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Time spent starting tablet: real 0.018s user 0.008s sys 0.007s
I20260620 18:31:42.223321 12188 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
I20260620 18:31:42.223461 12211 ts_tablet_manager.cc:1397] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 18:31:42.226914 12207 ts_tablet_manager.cc:1428] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Time spent starting tablet: real 0.025s user 0.010s sys 0.013s
I20260620 18:31:42.234905 12211 raft_consensus.cc:348] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.235484 12211 raft_consensus.cc:374] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:42.235652 12211 raft_consensus.cc:729] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f8054001974254916b4912753fe5f0, State: Initialized, Role: FOLLOWER
I20260620 18:31:42.236408 12211 consensus_queue.cc:260] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.237505 12054 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
I20260620 18:31:42.238276 12211 ts_tablet_manager.cc:1428] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260620 18:31:42.257444 12055 tablet.cc:2369] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:31:42.325109 11912 tablet.cc:2369] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:31:42.330615 12212 raft_consensus.cc:482] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:31:42.330974 12212 raft_consensus.cc:504] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.332690 12212 leader_election.cc:288] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a164ee49ea34cb19d456dc5d789aa9c (127.0.97.3:46187), 05f8054001974254916b4912753fe5f0 (127.0.97.2:40949)
W20260620 18:31:42.333681 12189 tablet.cc:2369] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:31:42.340811 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.333198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37361 (local address 127.0.97.3:46187)
0620 18:31:42.333411 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.333417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.333808 (+ 391us) server_negotiation.cc:407] Connection header received
0620 18:31:42.334219 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.334225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.334303 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.334371 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:31:42.335635 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.336543 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.338549 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.339806 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.339951 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.340043 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.340421 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.340566 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:31:42.340871 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.333058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46187 (local address 127.0.97.1:37361)
0620 18:31:42.333707 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.333729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.333965 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.334631 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.334634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.334656 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:31:42.335454 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.335466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.336676 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.336681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.338411 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.338423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.340048 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.340055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.340228 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.340265 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.340369 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.340530 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":74,"thread_start_us":316,"threads_started":1}
I20260620 18:31:42.341214 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.334151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58315 (local address 127.0.97.2:40949)
0620 18:31:42.334338 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.334346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.335221 (+ 875us) server_negotiation.cc:407] Connection header received
0620 18:31:42.335414 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.335421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.335500 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.335625 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:31:42.336155 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.336962 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.339149 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.340222 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.340373 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.340471 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.340904 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.341034 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:31:42.341379 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.333873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40949 (local address 127.0.97.1:58315)
0620 18:31:42.335097 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:42.335125 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:42.335285 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:42.335636 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:42.335641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:42.335664 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:31:42.336004 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.336016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.337120 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.337124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.338932 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:42.338948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.340542 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.340548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:42.340723 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.340756 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:31:42.340866 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:31:42.341035 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"spinlock_wait_cycles":18176,"thread_start_us":535,"threads_started":1}
I20260620 18:31:42.342298 12143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b719e893697542d6ac9160d718394e95" candidate_uuid: "117c3434df4c4817b7ee1638641ebe1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" is_pre_election: true
I20260620 18:31:42.342428 12009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b719e893697542d6ac9160d718394e95" candidate_uuid: "117c3434df4c4817b7ee1638641ebe1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f8054001974254916b4912753fe5f0" is_pre_election: true
I20260620 18:31:42.342790 12143 raft_consensus.cc:2454] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c3434df4c4817b7ee1638641ebe1a in term 0.
I20260620 18:31:42.343039 12009 raft_consensus.cc:2454] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c3434df4c4817b7ee1638641ebe1a in term 0.
I20260620 18:31:42.343529 11799 leader_election.cc:302] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117c3434df4c4817b7ee1638641ebe1a, 1a164ee49ea34cb19d456dc5d789aa9c; no voters:
I20260620 18:31:42.343953 12212 raft_consensus.cc:2781] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:31:42.344107 12212 raft_consensus.cc:482] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:31:42.344235 12212 raft_consensus.cc:3037] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:42.346416 12212 raft_consensus.cc:504] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.347054 12212 leader_election.cc:288] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [CANDIDATE]: Term 1 election: Requested vote from peers 1a164ee49ea34cb19d456dc5d789aa9c (127.0.97.3:46187), 05f8054001974254916b4912753fe5f0 (127.0.97.2:40949)
I20260620 18:31:42.347607 12143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b719e893697542d6ac9160d718394e95" candidate_uuid: "117c3434df4c4817b7ee1638641ebe1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a164ee49ea34cb19d456dc5d789aa9c"
I20260620 18:31:42.347683 12009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b719e893697542d6ac9160d718394e95" candidate_uuid: "117c3434df4c4817b7ee1638641ebe1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f8054001974254916b4912753fe5f0"
I20260620 18:31:42.347823 12143 raft_consensus.cc:3037] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:42.347856 12009 raft_consensus.cc:3037] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:42.351379 12143 raft_consensus.cc:2454] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c3434df4c4817b7ee1638641ebe1a in term 1.
I20260620 18:31:42.351379 12009 raft_consensus.cc:2454] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c3434df4c4817b7ee1638641ebe1a in term 1.
I20260620 18:31:42.351915 11799 leader_election.cc:302] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f8054001974254916b4912753fe5f0, 117c3434df4c4817b7ee1638641ebe1a; no voters:
I20260620 18:31:42.352198 12212 raft_consensus.cc:2781] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:31:42.352720 12212 raft_consensus.cc:686] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 1 LEADER]: Becoming Leader. State: Replica: 117c3434df4c4817b7ee1638641ebe1a, State: Running, Role: LEADER
I20260620 18:31:42.353215 12212 consensus_queue.cc:237] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } }
I20260620 18:31:42.357919 11698 catalog_manager.cc:5495] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 117c3434df4c4817b7ee1638641ebe1a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "117c3434df4c4817b7ee1638641ebe1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117c3434df4c4817b7ee1638641ebe1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39601 } health_report { overall_health: HEALTHY } } }
I20260620 18:31:42.408699 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.403495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.0.97.1:39601)
0620 18:31:42.403764 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.403769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.403784 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:31:42.403869 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.403873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.403952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.404106 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:31:42.404478 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.405203 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.406602 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.407003 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.407135 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.407622 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:31:42.408126 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:31:42.408296 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.408345 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.408424 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 18:31:42.417399 12009 raft_consensus.cc:1261] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 1 FOLLOWER]: Refusing update from remote peer 117c3434df4c4817b7ee1638641ebe1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:31:42.417398 12143 raft_consensus.cc:1261] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 1 FOLLOWER]: Refusing update from remote peer 117c3434df4c4817b7ee1638641ebe1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:31:42.418359 12218 consensus_queue.cc:1035] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a164ee49ea34cb19d456dc5d789aa9c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:31:42.418736 12212 consensus_queue.cc:1035] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f8054001974254916b4912753fe5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:31:42.433168 12222 mvcc.cc:205] Tried to move back new op lower bound from 7298991318690566144 to 7298991318442889216. Current Snapshot: MvccSnapshot[applied={T|T < 7298991318690566144}]
I20260620 18:31:42.446635 11698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:31:42.446918 11698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:31:42.450317 11698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cc711ab7639491db64f82df27821c9e: Sending DeleteTablet for 3 replicas of tablet b719e893697542d6ac9160d718394e95
I20260620 18:31:42.451574 12123 tablet_service.cc:1514] Processing DeleteTablet for tablet b719e893697542d6ac9160d718394e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:31:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50830
I20260620 18:31:42.451953 11846 tablet_service.cc:1514] Processing DeleteTablet for tablet b719e893697542d6ac9160d718394e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:31:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55160
I20260620 18:31:42.452678 11989 tablet_service.cc:1514] Processing DeleteTablet for tablet b719e893697542d6ac9160d718394e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:31:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55500
I20260620 18:31:42.455224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 11778
I20260620 18:31:42.461011 12233 tablet_replica.cc:330] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: stopping tablet replica
I20260620 18:31:42.462113 12233 raft_consensus.cc:2229] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:31:42.463270 12233 raft_consensus.cc:2258] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:31:42.464242 12231 tablet_replica.cc:330] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: stopping tablet replica
I20260620 18:31:42.466610 12233 ts_tablet_manager.cc:1905] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:31:42.467100 12231 raft_consensus.cc:2229] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:31:42.467736 12231 raft_consensus.cc:2258] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:31:42.475365 12233 ts_tablet_manager.cc:1918] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:31:42.476274 12232 tablet_replica.cc:330] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: stopping tablet replica
I20260620 18:31:42.476689 12232 raft_consensus.cc:2229] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:31:42.477010 12231 ts_tablet_manager.cc:1905] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:31:42.477358 12232 raft_consensus.cc:2258] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:31:42.479573 12232 ts_tablet_manager.cc:1905] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:31:42 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:31:42.483875 12231 ts_tablet_manager.cc:1918] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:31:42.484043 12231 log.cc:1196] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/b719e893697542d6ac9160d718394e95
I20260620 18:31:42.485018 12231 ts_tablet_manager.cc:1939] T b719e893697542d6ac9160d718394e95 P 1a164ee49ea34cb19d456dc5d789aa9c: Deleting consensus metadata
I20260620 18:31:42.486968 11682 catalog_manager.cc:4841] TS 1a164ee49ea34cb19d456dc5d789aa9c (127.0.97.3:46187): tablet b719e893697542d6ac9160d718394e95 (table test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]) successfully deleted
I20260620 18:31:42.487368 12233 log.cc:1196] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/b719e893697542d6ac9160d718394e95
I20260620 18:31:42.487942 12233 ts_tablet_manager.cc:1939] T b719e893697542d6ac9160d718394e95 P 05f8054001974254916b4912753fe5f0: Deleting consensus metadata
I20260620 18:31:42.487952 12232 ts_tablet_manager.cc:1918] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:31:42.488132 12232 log.cc:1196] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/b719e893697542d6ac9160d718394e95
I20260620 18:31:42.488651 12232 ts_tablet_manager.cc:1939] T b719e893697542d6ac9160d718394e95 P 117c3434df4c4817b7ee1638641ebe1a: Deleting consensus metadata
I20260620 18:31:42.489722 11682 catalog_manager.cc:4841] TS 05f8054001974254916b4912753fe5f0 (127.0.97.2:40949): tablet b719e893697542d6ac9160d718394e95 (table test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]) successfully deleted
I20260620 18:31:42.490229 11685 catalog_manager.cc:4841] TS 117c3434df4c4817b7ee1638641ebe1a (127.0.97.1:39601): tablet b719e893697542d6ac9160d718394e95 (table test-table [id=7d9cd0d3e7c04bc0ae4d8b893c889a08]) successfully deleted
I20260620 18:31:42.493265 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.457139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.97.1:39601)
0620 18:31:42.457358 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.457367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.468292 (+ 10925us) server_negotiation.cc:407] Connection header received
0620 18:31:42.468784 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.468794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.468903 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.469005 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:42.471329 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.473012 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.477746 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.478609 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.478749 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.484680 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:42.484705 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:42.484726 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:42.484785 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:42.489362 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.490990 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.491000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.491003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.491158 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.491683 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.491693 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.491695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.492057 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:42.492367 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.492680 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.492908 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:31:42.493948 11906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:42.595877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 11778
I20260620 18:31:42.614745 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 11917
Jun 20 18:31:42 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:31:42.629995 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.615506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.0.97.2:40949)
0620 18:31:42.615739 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.615747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.615805 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:31:42.616016 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.616031 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.616106 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.616216 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:42.617480 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.618231 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.619568 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.619956 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.620099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.624381 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:42.624395 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:42.624411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:42.624444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:42.627185 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.627953 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.627962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.627968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.628109 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.628489 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.628493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.628499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.628850 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:42.629159 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.629443 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.629648 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:31:42.630673 12049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:42.720811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 11917
I20260620 18:31:42.738909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 12060
Jun 20 18:31:42 dist-test-slave-mcnm krb5kdc[11222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:31:42.754801 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:42.739846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.0.97.3:46187)
0620 18:31:42.740023 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:42.740027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:42.740046 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:31:42.740114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:42.740117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:42.740181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:42.740279 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:42.741790 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.742578 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.744019 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:42.744475 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:42.744588 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:42.749119 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:42.749136 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:42.749154 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:42.749198 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:42.751996 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.752831 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.752839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.752842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.752979 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:42.753358 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:42.753361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:42.753363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:42.753735 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:42.753938 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:42.754353 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:31:42.754543 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:31:42.755502 12183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:42.844238 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 12060
I20260620 18:31:42.861815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 11666
I20260620 18:31:42.863013 11728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:42.974130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 11666
I20260620 18:31:42.996733 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:31:43.332134 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:31:43.332433 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:31:43.678174 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:31:43.678 UTC [11252] LOG: received smart shutdown request
2026-06-20 18:31:43.682 UTC [11252] LOG: background worker "logical replication launcher" (PID 11261) exited with exit code 1
2026-06-20 18:31:43.682 UTC [11256] LOG: shutting down
2026-06-20 18:31:43.866 UTC [11252] LOG: database system is shut down
2026-06-20T18:31:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48883 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:31:44 dist-test-slave-mcnm krb5kdc[12247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:31:44 dist-test-slave-mcnm krb5kdc[12247](info): set up 2 sockets
Jun 20 18:31:44 dist-test-slave-mcnm krb5kdc[12247](info): commencing operation
krb5kdc: starting...
W20260620 18:31:45.111960 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:31:45 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:31:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 18:31:45.430245 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/postgres -l logfile start
2026-06-20 18:31:47.997 UTC [12277] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:31:47.997 UTC [12277] LOG: listening on IPv4 address "127.0.97.20", port 44483
2026-06-20 18:31:48.001 UTC [12277] LOG: listening on Unix socket "/tmp/.s.PGSQL.44483"
2026-06-20 18:31:48.022 UTC [12280] LOG: database system was shut down at 2026-06-20 18:31:46 UTC
2026-06-20 18:31:48.026 UTC [12277] LOG: database system is ready to accept connections
I20260620 18:31:48.991781 388 mini_postgres.cc:96] Postgres bound to 44483
2026-06-20 18:31:49.009 UTC [12288] FATAL: database "slave" does not exist
127.0.97.20:44483 - accepting connections
I20260620 18:31:49.009687 388 mini_ranger.cc:161] Starting Ranger...
I20260620 18:31:49.035085 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:31:49.516752 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:31:49.516865 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin
2026-06-20 18:31:49,807 [I] DB FLAVOR :POSTGRES
2026-06-20 18:31:49,807 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:31:49,807 [I] Checking connection..
2026-06-20 18:31:49,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:31:50,066 [I] Checking connection passed.
2026-06-20 18:31:50,067 [I] --------- Verifying version history table ---------
2026-06-20 18:31:50,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:50,306 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:31:50,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:50,565 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:31:50,566 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:31:50,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:31:50.790 UTC [12352] WARNING: there is no transaction in progress
2026-06-20 18:31:50,797 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:31:50,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:31:51,051 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:31:51,051 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:31:51,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:31:51,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:31:51,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mcnm.c.gcp-upstream.internal','N') ;"
2026-06-20 18:31:51,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:31:52,015 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:31:52,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:31:52,255 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:31:52,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:31:52,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:31:52,751 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:31:52,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:31:53.751 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.768 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.784 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.802 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.826 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.844 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.902 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.909 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.924 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.934 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.945 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.958 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.968 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.974 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.981 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:53.989 UTC [12460] WARNING: there is no transaction in progress
2026-06-20 18:31:54,693 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:31:54,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:31:54,940 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:31:54,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:31:55,203 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:31:55,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:31:55,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:31:55,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:31:55,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:31:55,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mcnm.c.gcp-upstream.internal';"
2026-06-20 18:31:56,167 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:31:56,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:31:56,419 [I] DB_PATCHES have already been applied
I20260620 18:31:56.425418 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcfg4Mf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:31:56.425501 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 18:31:56.439951 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:53155
Jun 20, 2026 6:31:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:31:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:31:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:31:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:31:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:31:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:31:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:31:57.576663 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.567s user 0.001s sys 0.035s
Jun 20, 2026 6:32:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:32:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:32:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:32:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:32:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:32:19.766517 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:32:19 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:32:19.947546 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:32:19.970328 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:32:20.870565 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:32:20.870680 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms
I20260620 18:32:20.870733 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:44483
2026-06-20 18:32:21,221 [I] DB FLAVOR :POSTGRES
2026-06-20 18:32:21,221 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:32:21,221 [I] Checking connection
2026-06-20 18:32:21,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:32:21,467 [I] connection success
2026-06-20 18:32:21,468 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:32:21,468 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:32:21,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:32:21,705 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:32:21,706 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:32:21,706 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:32:21,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:32:21,965 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:32:22.470777 388 mini_ranger_kms.cc:310] Created kms service
I20260620 18:32:22.638422 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:32:22.680541 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:32:22 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:23.117408 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:32:23.117720 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcfg4Mf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcfg4Mf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:32:23.117794 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 18:32:23.132328 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12640
I20260620 18:32:23.132467 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:55559
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:32:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:32:25 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:25.466058 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.519s user 0.061s sys 0.308s
I20260620 18:32:25.466346 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:32:25.466564 388 mini_ranger_kms.cc:401] 127.0.97.20:55559/kms/v1/keys
Jun 20 18:32:25 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:32:26 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:32:28 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:32:28.408453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37553
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:28.742483 12691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:28.742844 12691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:28.743009 12691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:28.751223 12691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:28.751381 12691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:28.751500 12691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:28.751638 12691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:28.751734 12691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:28.764006 12691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:55559/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37553
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:28.765365 12691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:28.767097 12691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:28.780570 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:28.780601 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:28.780506 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:28.781709 12691 server_base.cc:1034] running on GCE node
I20260620 18:32:28.782567 12691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:28.784175 12691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:28.785421 12691 hybrid_clock.cc:648] HybridClock initialized: now 1781980348785370 us; error 47 us; skew 500 ppm
I20260620 18:32:28.785802 12691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:28 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:28.792861 12691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:28.794312 12691 webserver.cc:460] Webserver started at http://127.0.97.62:37713/ using document root <none> and password file <none>
I20260620 18:32:28.794914 12691 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:28.795008 12691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:28.795260 12691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:32:28 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:28.854779 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "be3b5edcb3354af6ba91169b019579c4"
format_stamp: "Formatted at 2026-06-20 18:32:28 on dist-test-slave-mcnm"
server_key: "92d750673f103ae228dd7d763eadeb82"
server_key_iv: "b89054259265170ef3c3dde0c16fe447"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:28.855758 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "be3b5edcb3354af6ba91169b019579c4"
format_stamp: "Formatted at 2026-06-20 18:32:28 on dist-test-slave-mcnm"
server_key: "92d750673f103ae228dd7d763eadeb82"
server_key_iv: "b89054259265170ef3c3dde0c16fe447"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:28.861690 12691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:28.882617 12706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:28.884625 12691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:32:28.884833 12691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "be3b5edcb3354af6ba91169b019579c4"
format_stamp: "Formatted at 2026-06-20 18:32:28 on dist-test-slave-mcnm"
server_key: "92d750673f103ae228dd7d763eadeb82"
server_key_iv: "b89054259265170ef3c3dde0c16fe447"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:28.885097 12691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:28.907404 12691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:28.912629 12691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:28.912997 12691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:28.934216 12691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37553
I20260620 18:32:28.934217 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37553 every 8 connection(s)
I20260620 18:32:28.936184 12691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:28.940711 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:28.944173 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 12691
I20260620 18:32:28.944418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:32:28.945322 388 external_mini_cluster.cc:1428] Setting key b8fd7a4d153a10c802f7575c1487c1a8
I20260620 18:32:28.958582 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Bootstrap starting.
I20260620 18:32:28.962942 12758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:28.964402 12758 log.cc:824] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:28.967988 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: No bootstrap required, opened a new log
Jun 20 18:32:28 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:28.977924 12758 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:28.978494 12758 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:28.978648 12758 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3b5edcb3354af6ba91169b019579c4, State: Initialized, Role: FOLLOWER
I20260620 18:32:28.979387 12758 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:28.979789 12758 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:28.979947 12758 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:28.980190 12758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:28.983321 12758 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:28.984094 12758 leader_election.cc:302] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3b5edcb3354af6ba91169b019579c4; no voters:
I20260620 18:32:28.984921 12758 leader_election.cc:288] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:28.984992 12763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:28.986075 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:28.947760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.0.97.62:37553)
0620 18:32:28.959381 (+ 11621us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:28.959400 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:28.960117 (+ 717us) server_negotiation.cc:407] Connection header received
0620 18:32:28.962749 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:28.962780 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:28.963383 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:28.963889 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:28.965640 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:28.967658 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:28.969619 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:28.970821 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:28.971038 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:28.976043 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:28.976056 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:28.976085 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:28.976150 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:28.980444 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:28.981412 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:28.981421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:28.981431 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:28.981619 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:28.982098 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:28.982103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:28.982107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:28.982864 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:28.983158 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:28.983768 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 18:32:28.984458 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11347,"thread_start_us":11125,"threads_started":1}
I20260620 18:32:28.989677 12763 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 LEADER]: Becoming Leader. State: Replica: be3b5edcb3354af6ba91169b019579c4, State: Running, Role: LEADER
I20260620 18:32:28.990129 12758 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:28.990463 12763 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:28.994228 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3b5edcb3354af6ba91169b019579c4. Latest consensus state: current_term: 1 leader_uuid: "be3b5edcb3354af6ba91169b019579c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260620 18:32:28.994709 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:28.995427 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3b5edcb3354af6ba91169b019579c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260620 18:32:28.996773 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:29.000736 12768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:29.011590 12768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:29.026356 12768 catalog_manager.cc:1347] Generated new cluster ID: 0dd3f70caaea40e0b6ed1d627038ef3d
I20260620 18:32:29.026537 12768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:29.042384 12768 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:29.044690 12768 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:29.055560 12768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Generated new TSK 0
I20260620 18:32:29.056116 12758 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:32:29.056589 12768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:32:29.855173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:32:30.261759 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:30.262137 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:30.262346 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:30.273847 12804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:30.274019 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:30.274219 12804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:32:30.287226 12804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:30.288600 12804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:30.290073 12804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:30.305836 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.306094 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.306443 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:30.306737 12804 server_base.cc:1034] running on GCE node
I20260620 18:32:30.307324 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:30.308346 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:30.309577 12804 hybrid_clock.cc:648] HybridClock initialized: now 1781980350309525 us; error 47 us; skew 500 ppm
I20260620 18:32:30.309900 12804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:30.313508 12804 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:32:30.320390 12804 webserver.cc:460] Webserver started at http://127.0.97.1:44623/ using document root <none> and password file <none>
I20260620 18:32:30.321399 12804 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:30.321552 12804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:30.321951 12804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:30.341195 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "ab2b178bbfdd4039ae01bc57a5d3d086"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-mcnm"
server_key: "f66e2c60d5088cacb691c8356667b48a"
server_key_iv: "8518e6417829c89c8b21774e3476ac5b"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:30.342291 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ab2b178bbfdd4039ae01bc57a5d3d086"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-mcnm"
server_key: "f66e2c60d5088cacb691c8356667b48a"
server_key_iv: "8518e6417829c89c8b21774e3476ac5b"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:30.355078 12804 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260620 18:32:30.390734 12824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.393186 12804 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.009s sys 0.009s
I20260620 18:32:30.393448 12804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "ab2b178bbfdd4039ae01bc57a5d3d086"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-mcnm"
server_key: "f66e2c60d5088cacb691c8356667b48a"
server_key_iv: "8518e6417829c89c8b21774e3476ac5b"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:30.393798 12804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:30.463765 12804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:30.468043 12804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:30.468364 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:30.476059 12804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:30.477645 12804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:30.477757 12804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.477888 12804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:30.477975 12804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.561123 12804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42583
I20260620 18:32:30.561148 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42583 every 8 connection(s)
I20260620 18:32:30.562500 12804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:30.563122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 12804
I20260620 18:32:30.563359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:30.564466 388 external_mini_cluster.cc:1428] Setting key dc44064aff22a6869cbbe21f4c4d9ea0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:30.624123 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.580654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44273 (local address 127.0.97.62:37553)
0620 18:32:30.591865 (+ 11211us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:30.591871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:30.591896 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:30.602561 (+ 10665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:30.602569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:30.602681 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:30.602788 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:30.605045 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.606263 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.608721 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.609395 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.609496 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.614352 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:30.614359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:30.614364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:30.614406 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:30.618066 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.619331 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.619336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.619339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.619504 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.621907 (+ 2403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.621912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.621916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.622315 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:30.622454 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:30.623464 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0620 18:32:30.623720 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11081,"thread_start_us":10966,"threads_started":1}
I20260620 18:32:30.626899 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.580002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.1:44273)
0620 18:32:30.591359 (+ 11357us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:30.591418 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:30.602027 (+ 10609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:30.603275 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:30.603289 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:30.603945 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:30.604828 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.604855 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.606474 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.606480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.608546 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.608561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.609916 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.609923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.610224 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.611018 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:30.611048 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:30.614069 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:30.618276 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.618287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.618306 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.618929 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.620786 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.620791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.620795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.621054 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.622557 (+ 1503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:30.622568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:30.622999 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 18:32:30.624676 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0620 18:32:30.625249 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11002,"thread_start_us":10773,"threads_started":1}
I20260620 18:32:30.629215 12937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:30.629544 12937 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:30.630446 12937 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:30.633045 12723 ts_manager.cc:194] Registered new tserver with Master: ab2b178bbfdd4039ae01bc57a5d3d086 (127.0.97.1:42583)
I20260620 18:32:30.635193 12723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:32:30.648602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:32:30 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 18:32:31.132479 12943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:31.132977 12943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:31.133203 12943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:31.144732 12943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:31.144908 12943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:31.145161 12943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:32:31.160878 12943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:31.162823 12943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:31.164690 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:31.200459 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:31.200945 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:31.211668 12943 server_base.cc:1034] running on GCE node
W20260620 18:32:31.211922 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:31.212677 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:31.213791 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:31.215066 12943 hybrid_clock.cc:648] HybridClock initialized: now 1781980351214998 us; error 60 us; skew 500 ppm
I20260620 18:32:31.215420 12943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:31.218884 12943 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:32:31.224910 12943 webserver.cc:460] Webserver started at http://127.0.97.2:44003/ using document root <none> and password file <none>
I20260620 18:32:31.225632 12943 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:31.225787 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:31.226111 12943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:31.249903 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "165b317e01ed4ff8a0d42b8123f787f7"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "0c70d54ff14df648700ba28407268935"
server_key_iv: "c56efdc7cf38e6f988e8322e873f2141"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.250916 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "165b317e01ed4ff8a0d42b8123f787f7"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "0c70d54ff14df648700ba28407268935"
server_key_iv: "c56efdc7cf38e6f988e8322e873f2141"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.256639 12943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:31.272564 12967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.274111 12943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:32:31.274329 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "165b317e01ed4ff8a0d42b8123f787f7"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "0c70d54ff14df648700ba28407268935"
server_key_iv: "c56efdc7cf38e6f988e8322e873f2141"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.274595 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:31.296298 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:31.300189 12943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:31.300509 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:31.301530 12943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:31.302856 12943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:31.302942 12943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.303078 12943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:31.303128 12943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.333833 12943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41487
I20260620 18:32:31.333881 13079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41487 every 8 connection(s)
I20260620 18:32:31.335067 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:31.337445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 12943
I20260620 18:32:31.337584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:31.338217 388 external_mini_cluster.cc:1428] Setting key 265aff65db67dc625a2188ae2d0ca31f
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:31.357295 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.337803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52581 (local address 127.0.97.62:37553)
0620 18:32:31.340692 (+ 2889us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.340696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.340712 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:31.341628 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.341636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.341722 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.341845 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.343166 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.344400 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.345926 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.346664 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.346802 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.350062 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.350071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.350079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.350123 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.353903 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.354569 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.354577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.354584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.354725 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.355302 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.355309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.355316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.355721 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.355890 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.356508 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.356768 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2777,"thread_start_us":2659,"threads_started":1}
I20260620 18:32:31.357805 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.338303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.2:52581)
0620 18:32:31.339188 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.339234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.341349 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.342029 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.342037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.342464 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:31.343000 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.343016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.344556 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.344560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.345780 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.345790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.346877 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.346883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.347074 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.347766 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:31.347791 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:31.349853 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:31.354040 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.354046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.354058 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.354453 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.354866 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.354870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.354872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.355173 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.355869 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:31.355875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:31.356156 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.356379 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.356719 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260620 18:32:31.359197 13080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:31.359565 13080 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:31.360457 13080 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:31.362273 12723 ts_manager.cc:194] Registered new tserver with Master: 165b317e01ed4ff8a0d42b8123f787f7 (127.0.97.2:41487)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:31.363222 12723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:32:31.391394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:31.639195 12937 heartbeater.cc:503] Master 127.0.97.62:37553 was elected leader, sending a full tablet report...
W20260620 18:32:31.716874 13086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:31.717178 13086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:31.717341 13086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:31.724781 13086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:31.724920 13086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:31.725098 13086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:32:31.734520 13086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:31.735834 13086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:31.737186 13086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:31.750442 13091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:31.750535 13092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:31.750429 13094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:31.751226 13086 server_base.cc:1034] running on GCE node
I20260620 18:32:31.751868 13086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:31.752737 13086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:31.753978 13086 hybrid_clock.cc:648] HybridClock initialized: now 1781980351753920 us; error 53 us; skew 500 ppm
I20260620 18:32:31.754292 13086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:31.757525 13086 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:32:31.758973 13086 webserver.cc:460] Webserver started at http://127.0.97.3:43001/ using document root <none> and password file <none>
I20260620 18:32:31.759757 13086 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:31.759902 13086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:31.760203 13086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:31.775408 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "c9f230e9712c42b9a5ab649cace56cb2"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "ca29ddee1b79fde7d2381e28452a16a9"
server_key_iv: "755a5855c8f9a2c15e64327d896bc9af"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.776314 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c9f230e9712c42b9a5ab649cace56cb2"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "ca29ddee1b79fde7d2381e28452a16a9"
server_key_iv: "755a5855c8f9a2c15e64327d896bc9af"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.781809 13086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:31.798462 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.800252 13086 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:31.800433 13086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "c9f230e9712c42b9a5ab649cace56cb2"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "ca29ddee1b79fde7d2381e28452a16a9"
server_key_iv: "755a5855c8f9a2c15e64327d896bc9af"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:31.800666 13086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:31.845041 13086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:31.849135 13086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:31.849444 13086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:31.850459 13086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:31.852012 13086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:31.852114 13086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.852260 13086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:31.852334 13086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.885833 13086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39995
I20260620 18:32:31.885895 13213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39995 every 8 connection(s)
I20260620 18:32:31.887125 13086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:31.894703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13086
I20260620 18:32:31.894966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:31.895577 388 external_mini_cluster.cc:1428] Setting key e003f7c43153d7cdf81234026f003c83
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:31.908593 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.889810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40289 (local address 127.0.97.62:37553)
0620 18:32:31.890313 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.890318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.891238 (+ 920us) server_negotiation.cc:407] Connection header received
0620 18:32:31.893236 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.893243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.893344 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.893464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.895028 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.896208 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.898182 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.898863 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.899051 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.902727 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.902732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.902736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.902771 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.905625 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.906432 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.906435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.906438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.906561 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.907016 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.907020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.907022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.907293 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.907404 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.908165 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.908338 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":295,"threads_started":1}
I20260620 18:32:31.909624 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.890264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.3:40289)
0620 18:32:31.891088 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.891134 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.892946 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.893698 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.893706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.894115 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:31.894818 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.894843 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.896390 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.896398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.898027 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.898040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.899170 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.899176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.899382 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.900128 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:31.900153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:31.902457 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:31.905795 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.905803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.905817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.906321 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.906688 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.906692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.906694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.906894 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.907410 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:31.907416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:31.907809 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.908061 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.908493 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":348,"threads_started":1}
I20260620 18:32:31.911096 13214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:31.911412 13214 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:31.912117 13214 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:31.913417 12723 ts_manager.cc:194] Registered new tserver with Master: c9f230e9712c42b9a5ab649cace56cb2 (127.0.97.3:39995)
I20260620 18:32:31.914209 12723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40289
I20260620 18:32:31.923663 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:31.925289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 12804
Jun 20 18:32:31 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:32:31.942440 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.926318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.97.1:42583)
0620 18:32:31.926927 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.926934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.926958 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:31.927014 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.927020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.927217 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.927337 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.928818 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.929816 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.931269 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.931866 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.931968 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.936540 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.936557 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.936572 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.936618 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.939699 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.940343 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.940352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.940356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.940510 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.940872 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.940878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.940882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.941214 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.941525 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.941822 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.942099 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
I20260620 18:32:31.944011 12932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:32.027586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 12804
I20260620 18:32:32.043340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 12943
Jun 20 18:32:32 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:32:32.059078 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:32.044345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.97.2:41487)
0620 18:32:32.044919 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:32.044926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:32.044950 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:32.045011 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:32.045018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:32.045246 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:32.045375 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:32.046676 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:32.047759 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:32.048978 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:32.049515 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:32.049609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:32.053810 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:32.053820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:32.053834 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:32.053866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:32.056585 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:32.057212 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:32.057218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:32.057221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:32.057337 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:32.057741 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:32.057744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:32.057746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:32.058074 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:32.058308 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:32.058591 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:32:32.058776 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260620 18:32:32.060524 13075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:32.143129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 12943
I20260620 18:32:32.158820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13086
Jun 20 18:32:32 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:32:32.174469 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:32.159874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.97.3:39995)
0620 18:32:32.160485 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:32.160492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:32.160518 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:32.160576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:32.160582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:32.160798 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:32.160924 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:32.162158 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:32.163151 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:32.164470 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:32.165018 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:32.165113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:32.169266 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:32.169275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:32.169289 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:32.169323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:32.171998 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:32.172624 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:32.172630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:32.172632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:32.172750 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:32.173112 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:32.173115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:32.173117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:32.173419 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:32.173641 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:32.173895 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:32:32.174088 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":342,"threads_started":1}
I20260620 18:32:32.175743 13209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:32.257753 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13086
I20260620 18:32:32.273566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 12691
I20260620 18:32:32.274863 12753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:32.381541 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 12691
I20260620 18:32:32.402297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--webserver_interface=127.0.97.62
--webserver_port=37713
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37553
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:32.740856 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:32.741194 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:32.741356 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:32.748708 13226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:32.748850 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:32.748957 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:32.749049 13226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:32.749133 13226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:32.759466 13226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37553
--ranger_config_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:32.760973 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:32.762799 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:32.777133 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:32.777088 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:32.777184 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:32.778141 13226 server_base.cc:1034] running on GCE node
I20260620 18:32:32.778888 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:32.780349 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:32.781585 13226 hybrid_clock.cc:648] HybridClock initialized: now 1781980352781549 us; error 51 us; skew 500 ppm
I20260620 18:32:32.781983 13226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:32 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:32.785140 13226 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:32.786588 13226 webserver.cc:460] Webserver started at http://127.0.97.62:37713/ using document root <none> and password file <none>
I20260620 18:32:32.787180 13226 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:32.787281 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:32:32 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:32.869993 13226 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:32:32.873003 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:32.874534 13226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:32.874750 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "be3b5edcb3354af6ba91169b019579c4"
format_stamp: "Formatted at 2026-06-20 18:32:28 on dist-test-slave-mcnm"
server_key: "92d750673f103ae228dd7d763eadeb82"
server_key_iv: "b89054259265170ef3c3dde0c16fe447"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:32.875829 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:32.928032 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:32.932161 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:32.932480 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:32.951773 13226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37553
I20260620 18:32:32.951802 13293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37553 every 8 connection(s)
I20260620 18:32:32.953272 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:32.956673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13226
I20260620 18:32:32.957509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42583
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37553
--webserver_port=44623
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:32.960407 13294 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:32:32.963668 13294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Bootstrap starting.
I20260620 18:32:32.991228 13294 log.cc:824] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:33.002190 13294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:33.002858 13294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Bootstrap complete.
I20260620 18:32:33.013391 13294 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:33.014297 13294 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3b5edcb3354af6ba91169b019579c4, State: Initialized, Role: FOLLOWER
I20260620 18:32:33.015004 13294 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:33.015250 13294 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:33.015386 13294 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:33.015591 13294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:32:33.019263 13294 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:33.019824 13294 leader_election.cc:302] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3b5edcb3354af6ba91169b019579c4; no voters:
I20260620 18:32:33.020491 13294 leader_election.cc:288] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:32:33.020726 13298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:32:33.021889 13298 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [term 2 LEADER]: Becoming Leader. State: Replica: be3b5edcb3354af6ba91169b019579c4, State: Running, Role: LEADER
I20260620 18:32:33.022459 13298 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260620 18:32:33.022611 13294 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:33.025218 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3b5edcb3354af6ba91169b019579c4. Latest consensus state: current_term: 2 leader_uuid: "be3b5edcb3354af6ba91169b019579c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260620 18:32:33.025695 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:33.026309 13299 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be3b5edcb3354af6ba91169b019579c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3b5edcb3354af6ba91169b019579c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260620 18:32:33.026646 13299 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:33.036592 13305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:33.043651 13305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:33.045509 13305 catalog_manager.cc:1259] Loaded cluster ID: 0dd3f70caaea40e0b6ed1d627038ef3d
I20260620 18:32:33.045711 13305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:33.051019 13305 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:33.054416 13305 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Loaded TSK: 0
I20260620 18:32:33.055179 13294 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:32:33.056365 13305 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:32:33.426870 13296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:33.427356 13296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:33.427623 13296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:33.438349 13296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:33.438566 13296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:33.438824 13296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:32:33.453763 13296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44623
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:33.455818 13296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:33.457635 13296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:33.498529 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.499632 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.513296 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:33.517282 13296 server_base.cc:1034] running on GCE node
I20260620 18:32:33.518059 13296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:33.524593 13296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:33.527348 13296 hybrid_clock.cc:648] HybridClock initialized: now 1781980353527272 us; error 73 us; skew 500 ppm
I20260620 18:32:33.527774 13296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:33 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:33.531215 13296 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:32:33.537215 13296 webserver.cc:460] Webserver started at http://127.0.97.1:44623/ using document root <none> and password file <none>
I20260620 18:32:33.538110 13296 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:33.538271 13296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:32:33 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:33.594029 13296 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.004s
I20260620 18:32:33.598861 13342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.600869 13296 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 18:32:33.601089 13296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "ab2b178bbfdd4039ae01bc57a5d3d086"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-mcnm"
server_key: "f66e2c60d5088cacb691c8356667b48a"
server_key_iv: "8518e6417829c89c8b21774e3476ac5b"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:33.602694 13296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:33.675127 13296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:33.680552 13296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:33.680869 13296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:33.681995 13296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:33.683804 13296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:33.683919 13296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.684026 13296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:33.684096 13296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.762543 13296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42583
I20260620 18:32:33.762651 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42583 every 8 connection(s)
I20260620 18:32:33.764181 13296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:33.771881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13296
I20260620 18:32:33.772709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41487
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37553
--webserver_port=44003
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:33 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:33.808899 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.768086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.1:58487)
0620 18:32:33.772390 (+ 4304us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:33.772446 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:33.774799 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:33.778130 (+ 3331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:33.778141 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:33.778775 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:33.779643 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:33.779707 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:33.781431 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:33.781437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:33.782790 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:33.782804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:33.784439 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:33.784449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:33.784761 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:33.785578 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:33.785609 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:33.788609 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:33.802660 (+ 14051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:33.802671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:33.802691 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:33.803405 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:33.804311 (+ 906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:33.804316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:33.804320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:33.804608 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:33.805779 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:33.805788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:33.806252 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 18:32:33.806566 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 18:32:33.807183 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3987,"thread_start_us":11202,"threads_started":1}
I20260620 18:32:33.812029 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.769208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58487 (local address 127.0.97.62:37553)
0620 18:32:33.775776 (+ 6568us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:33.775795 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:33.775842 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:32:33.776324 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:33.776345 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:33.776885 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:33.777376 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:33.779926 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:33.781254 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:33.783054 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:33.784129 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:33.784391 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:33.788986 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:33.788999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:33.789020 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:33.789077 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:33.802379 (+ 13302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:33.803796 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:33.803809 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:33.803819 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:33.804152 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:33.804802 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:33.804807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:33.804810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:33.805552 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:33.809964 (+ 4412us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:33.810252 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:32:33.810685 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6255,"thread_start_us":10905,"threads_started":1}
I20260620 18:32:33.815938 13456 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:33.816277 13456 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:33.817219 13456 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
W20260620 18:32:33.821045 13456 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37553 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:32:33.821290 13456 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:33.821611 13456 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:33.823014 13456 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:33.823396 13456 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
W20260620 18:32:33.824664 13456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:32:34.194276 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:34.194593 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:34.194767 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:34.203087 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:34.203230 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:34.203445 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:32:34.216502 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44003
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:34.218389 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:34.220185 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:34.260584 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:34.263377 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:34.263897 13460 server_base.cc:1034] running on GCE node
W20260620 18:32:34.264954 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:34.265844 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:34.275923 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:34.277314 13460 hybrid_clock.cc:648] HybridClock initialized: now 1781980354277207 us; error 122 us; skew 500 ppm
I20260620 18:32:34.277838 13460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:34.282420 13460 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:32:34.284539 13460 webserver.cc:460] Webserver started at http://127.0.97.2:44003/ using document root <none> and password file <none>
I20260620 18:32:34.285526 13460 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:34.285692 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:34.309087 13460 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:34.313086 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:34.315191 13460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 18:32:34.315471 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "165b317e01ed4ff8a0d42b8123f787f7"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "0c70d54ff14df648700ba28407268935"
server_key_iv: "c56efdc7cf38e6f988e8322e873f2141"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:34.317138 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:34.344954 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:34.350086 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:34.350389 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:34.351605 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:34.362864 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:34.362989 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:34.363111 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:34.363188 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:34.431483 13460 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41487
I20260620 18:32:34.431592 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41487 every 8 connection(s)
I20260620 18:32:34.433336 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:34.441227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13460
I20260620 18:32:34.441968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39995
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37553
--webserver_port=43001
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:34.461531 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.437663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59819 (local address 127.0.97.62:37553)
0620 18:32:34.439379 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.439385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.439411 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:34.441678 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.441684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.441775 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.441915 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.443784 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:34.444762 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:34.446353 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:34.446809 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:34.446956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:34.450655 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.450663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.450668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:34.450715 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.457043 (+ 6328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:34.458377 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:34.458382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:34.458386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:34.458551 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:34.459206 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:34.459212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:34.459216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:34.459632 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.459790 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.460798 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.461065 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1582,"spinlock_wait_cycles":34560,"thread_start_us":1561,"threads_started":1}
I20260620 18:32:34.462869 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.436954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.2:59819)
0620 18:32:34.438808 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.438857 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.441350 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.442192 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.442204 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.442688 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.443506 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:34.443567 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:34.444943 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:34.444947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:34.446193 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:34.446204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:34.447047 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:34.447057 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:34.447323 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:34.448103 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:34.448126 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.450250 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.457281 (+ 7031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:34.457292 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:34.457313 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:34.458171 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:34.458705 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:34.458714 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:34.458720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:34.459017 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:34.459781 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.459792 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.460298 (+ 506us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.460639 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.461300 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":1228,"threads_started":1}
I20260620 18:32:34.464599 13601 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:34.464954 13601 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:34.465924 13601 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:34.468220 13259 ts_manager.cc:194] Registered new tserver with Master: 165b317e01ed4ff8a0d42b8123f787f7 (127.0.97.2:41487)
I20260620 18:32:34.470476 13259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59819
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:34.824844 13456 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:34.825285 13456 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:34.832052 13259 ts_manager.cc:194] Registered new tserver with Master: ab2b178bbfdd4039ae01bc57a5d3d086 (127.0.97.1:42583)
I20260620 18:32:34.833181 13259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58487
W20260620 18:32:34.909906 13605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:34.910297 13605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:34.910439 13605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:34.921705 13605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:34.921849 13605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:34.922014 13605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:32:34.937624 13605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43001
--tserver_master_addrs=127.0.97.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:34.939325 13605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:34.941046 13605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:34.958659 13610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:34.969961 13614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:34.970489 13611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:34.970925 13605 server_base.cc:1034] running on GCE node
I20260620 18:32:34.971845 13605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:34.976655 13605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:34.977918 13605 hybrid_clock.cc:648] HybridClock initialized: now 1781980354977843 us; error 69 us; skew 500 ppm
I20260620 18:32:34.978372 13605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:34.982985 13605 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:32:34.993006 13605 webserver.cc:460] Webserver started at http://127.0.97.3:43001/ using document root <none> and password file <none>
I20260620 18:32:34.994115 13605 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:34.994809 13605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:32:34 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 18:32:35 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 18:32:35.023257 13605 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:35.027184 13622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:35.036530 13605 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260620 18:32:35.036898 13605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "c9f230e9712c42b9a5ab649cace56cb2"
format_stamp: "Formatted at 2026-06-20 18:32:31 on dist-test-slave-mcnm"
server_key: "ca29ddee1b79fde7d2381e28452a16a9"
server_key_iv: "755a5855c8f9a2c15e64327d896bc9af"
server_key_version: "kuduclusterkey@0"
I20260620 18:32:35.038667 13605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:35.100160 13605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:35.105638 13605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:35.106012 13605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:35.113644 13605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:35.136487 13605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:35.136714 13605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:32:35.136951 13605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:35.137076 13605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:35.305346 13605 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39995
I20260620 18:32:35.305418 13734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39995 every 8 connection(s)
I20260620 18:32:35.307392 13605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:35.316174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13605
Jun 20 18:32:35 dist-test-slave-mcnm krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:35.348181 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.317475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.97.62:37553)
0620 18:32:35.329759 (+ 12284us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.329765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.329783 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:35.329838 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.329843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.329902 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.330017 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.331957 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.333148 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.336925 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.337610 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.337734 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.340828 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.340835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.340842 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.340887 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.345080 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.345884 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.345890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.345896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.346060 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.346516 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.346522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.346525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.346900 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.347178 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.347501 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.347756 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12158,"thread_start_us":10461,"threads_started":1}
I20260620 18:32:35.351488 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.311980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37553 (local address 127.0.97.3:59765)
0620 18:32:35.312867 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.312915 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.315247 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.328863 (+ 13616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.328878 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.329664 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:35.330637 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.330668 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.332095 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.332102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.334122 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.334135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.334943 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.334950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.335185 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.337271 (+ 2086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:35.337308 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:35.340968 (+ 3660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:35.345679 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:35.345689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:35.345708 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:35.346580 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:35.346984 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:35.346988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:35.346991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:35.347266 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:35.348893 (+ 1627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:35.348902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:35.349324 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.349634 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.350213 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":474,"threads_started":1}
I20260620 18:32:35.352280 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.312465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59765 (local address 127.0.97.62:37553)
0620 18:32:35.328135 (+ 15670us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.328142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.328170 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:35.328246 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.328251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.328343 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.328498 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.330862 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.331890 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.334279 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.334695 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.334834 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.341256 (+ 6422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.341262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.341267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.341331 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.345505 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.346720 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.346724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.346728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.346839 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.348374 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.348381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.348386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.348735 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.348902 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.351728 (+ 2826us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.351940 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15501,"thread_start_us":17040,"threads_started":1}
I20260620 18:32:35.353423 13735 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37553
I20260620 18:32:35.353781 13735 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:35.354492 13735 heartbeater.cc:511] Master 127.0.97.62:37553 requested a full tablet report, sending...
I20260620 18:32:35.356009 13259 ts_manager.cc:194] Registered new tserver with Master: c9f230e9712c42b9a5ab649cace56cb2 (127.0.97.3:39995)
I20260620 18:32:35.357148 13259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59765
I20260620 18:32:35.363627 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:35.382438 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.367010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.97.62:37553)
0620 18:32:35.367237 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.367247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.367720 (+ 473us) server_negotiation.cc:407] Connection header received
0620 18:32:35.367887 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.367890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.367962 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.368045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.370017 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.370850 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.372895 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.373316 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.373414 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.376802 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.376807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.376810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.376841 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.379750 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.380493 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.380502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.380505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.380646 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.381042 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.381051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.381053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.381334 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.381589 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.381906 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.382124 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:32:35.387763 13259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:35.395241 13259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:35.424047 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.410442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58268)
0620 18:32:35.411474 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.411538 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.411718 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.413019 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.413030 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.413071 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:35.413586 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.413603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.418822 (+ 5219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.418828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.420993 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.421008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.422906 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.422913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.423170 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.423234 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.423454 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.423676 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":687,"threads_started":1}
I20260620 18:32:35.424045 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.410910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.0.97.1:42583)
0620 18:32:35.412243 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.412251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.412280 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:32:35.412405 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.412417 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.412736 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.413863 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:35.414238 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.418659 (+ 4421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.421197 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.422687 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.422836 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.423072 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.423535 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.423710 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":1196,"threads_started":1}
I20260620 18:32:35.426141 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.410877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39995 (local address 127.0.0.1:53970)
0620 18:32:35.416836 (+ 5959us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.416861 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.417000 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.418329 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.418336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.418368 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:35.418782 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.418794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.420632 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.420640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.422568 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.422580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.425009 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.425014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.425722 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.425745 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.425819 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.425903 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"thread_start_us":5621,"threads_started":1}
I20260620 18:32:35.426402 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.411030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.0.97.3:39995)
0620 18:32:35.417670 (+ 6640us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.417683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.417721 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:32:35.417839 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.417849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.418121 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.418316 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:35.418981 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.420488 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.422717 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.424582 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.424698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.425554 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.425908 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.426064 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6416,"thread_start_us":6237,"threads_started":1}
I20260620 18:32:35.428711 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.411990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41487 (local address 127.0.0.1:41444)
0620 18:32:35.418308 (+ 6318us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.418324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.418527 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.419428 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.419432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.419464 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:35.419920 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.419931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.421832 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.421838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.423956 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.423969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.427931 (+ 3962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.427940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.428141 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.428182 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.428269 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.428395 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6187,"thread_start_us":6067,"threads_started":1}
I20260620 18:32:35.428834 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.412303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.97.2:41487)
0620 18:32:35.417919 (+ 5616us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.417925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.418418 (+ 493us) server_negotiation.cc:407] Connection header received
0620 18:32:35.418742 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.418752 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.418989 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.419150 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:35.420115 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.421682 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.424142 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.427719 (+ 3577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.427848 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.428020 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.428360 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.428527 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5469,"thread_start_us":5362,"threads_started":1}
I20260620 18:32:35.428293 13391 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35bff5081d4a3db06af0aacf496284 (DEFAULT_TABLE table=test-table [id=3c4f1614046f46a0a5ba9aa780a80c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:35.429855 13391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35bff5081d4a3db06af0aacf496284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:35.429636 13670 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35bff5081d4a3db06af0aacf496284 (DEFAULT_TABLE table=test-table [id=3c4f1614046f46a0a5ba9aa780a80c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:35.430955 13670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35bff5081d4a3db06af0aacf496284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:35.432308 13536 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35bff5081d4a3db06af0aacf496284 (DEFAULT_TABLE table=test-table [id=3c4f1614046f46a0a5ba9aa780a80c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:35.433614 13536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35bff5081d4a3db06af0aacf496284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:35.445607 13763 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Bootstrap starting.
I20260620 18:32:35.448678 13763 tablet_bootstrap.cc:654] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:35.449527 13764 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Bootstrap starting.
I20260620 18:32:35.450361 13763 log.cc:824] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:35.451110 13765 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Bootstrap starting.
I20260620 18:32:35.453752 13765 tablet_bootstrap.cc:654] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:35.454618 13764 tablet_bootstrap.cc:654] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:35.454958 13765 log.cc:824] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:35.455796 13764 log.cc:824] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:35.456593 13763 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: No bootstrap required, opened a new log
I20260620 18:32:35.456856 13763 ts_tablet_manager.cc:1397] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 18:32:35.457849 13765 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: No bootstrap required, opened a new log
I20260620 18:32:35.458094 13765 ts_tablet_manager.cc:1397] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 18:32:35.458689 13764 tablet_bootstrap.cc:492] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: No bootstrap required, opened a new log
I20260620 18:32:35.458922 13764 ts_tablet_manager.cc:1397] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 18:32:35.464444 13763 raft_consensus.cc:348] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.464870 13763 raft_consensus.cc:374] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:35.464959 13763 raft_consensus.cc:729] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2b178bbfdd4039ae01bc57a5d3d086, State: Initialized, Role: FOLLOWER
I20260620 18:32:35.465535 13763 consensus_queue.cc:260] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.468199 13456 heartbeater.cc:503] Master 127.0.97.62:37553 was elected leader, sending a full tablet report...
I20260620 18:32:35.468464 13765 raft_consensus.cc:348] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.468838 13763 ts_tablet_manager.cc:1428] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 18:32:35.468866 13765 raft_consensus.cc:374] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:35.468958 13765 raft_consensus.cc:729] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165b317e01ed4ff8a0d42b8123f787f7, State: Initialized, Role: FOLLOWER
I20260620 18:32:35.469081 13764 raft_consensus.cc:348] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.469465 13764 raft_consensus.cc:374] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:35.469571 13764 raft_consensus.cc:729] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f230e9712c42b9a5ab649cace56cb2, State: Initialized, Role: FOLLOWER
I20260620 18:32:35.469535 13765 consensus_queue.cc:260] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.470250 13764 consensus_queue.cc:260] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.470582 13601 heartbeater.cc:503] Master 127.0.97.62:37553 was elected leader, sending a full tablet report...
I20260620 18:32:35.471357 13765 ts_tablet_manager.cc:1428] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260620 18:32:35.472424 13735 heartbeater.cc:503] Master 127.0.97.62:37553 was elected leader, sending a full tablet report...
I20260620 18:32:35.473111 13764 ts_tablet_manager.cc:1428] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260620 18:32:35.494863 13769 raft_consensus.cc:482] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:35.495168 13769 raft_consensus.cc:504] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.496930 13769 leader_election.cc:288] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f230e9712c42b9a5ab649cace56cb2 (127.0.97.3:39995), 165b317e01ed4ff8a0d42b8123f787f7 (127.0.97.2:41487)
I20260620 18:32:35.503504 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.497395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39619 (local address 127.0.97.3:39995)
0620 18:32:35.497656 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.497665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.498338 (+ 673us) server_negotiation.cc:407] Connection header received
0620 18:32:35.498546 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.498550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.498606 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.498713 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:35.499289 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.500148 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.501849 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.502612 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.502757 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.502859 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.503129 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.503246 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 18:32:35.503687 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.497363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39995 (local address 127.0.97.1:39619)
0620 18:32:35.498213 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.498236 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.498405 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.498734 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.498739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.498774 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:35.499146 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.499157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.500269 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.500276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.501704 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.501715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.502810 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.502816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.502971 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.502994 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.503108 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.503233 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":20,"thread_start_us":491,"threads_started":1}
I20260620 18:32:35.503863 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.497393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39461 (local address 127.0.97.2:41487)
0620 18:32:35.497651 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.497660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.498633 (+ 973us) server_negotiation.cc:407] Connection header received
0620 18:32:35.498826 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.498830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.498896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.498991 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:35.499586 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.500436 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.502122 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.502869 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.503014 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.503112 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.503394 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.503537 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 18:32:35.504134 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.497389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41487 (local address 127.0.97.1:39461)
0620 18:32:35.498531 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.498546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.498688 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.499008 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.499012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.499033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:35.499396 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.499404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.500578 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.500585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.501957 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.501969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.503096 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.503102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.503239 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.503261 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:35.503369 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:32:35.503486 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":3,"thread_start_us":816,"threads_started":1}
I20260620 18:32:35.504755 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35bff5081d4a3db06af0aacf496284" candidate_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f230e9712c42b9a5ab649cace56cb2" is_pre_election: true
I20260620 18:32:35.505033 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35bff5081d4a3db06af0aacf496284" candidate_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165b317e01ed4ff8a0d42b8123f787f7" is_pre_election: true
I20260620 18:32:35.505220 13690 raft_consensus.cc:2454] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab2b178bbfdd4039ae01bc57a5d3d086 in term 0.
I20260620 18:32:35.505577 13556 raft_consensus.cc:2454] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab2b178bbfdd4039ae01bc57a5d3d086 in term 0.
I20260620 18:32:35.505903 13344 leader_election.cc:302] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab2b178bbfdd4039ae01bc57a5d3d086, c9f230e9712c42b9a5ab649cace56cb2; no voters:
I20260620 18:32:35.506313 13769 raft_consensus.cc:2781] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:35.506454 13769 raft_consensus.cc:482] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:35.506559 13769 raft_consensus.cc:3037] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:35.508833 13769 raft_consensus.cc:504] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.509474 13769 leader_election.cc:288] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [CANDIDATE]: Term 1 election: Requested vote from peers c9f230e9712c42b9a5ab649cace56cb2 (127.0.97.3:39995), 165b317e01ed4ff8a0d42b8123f787f7 (127.0.97.2:41487)
I20260620 18:32:35.509779 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35bff5081d4a3db06af0aacf496284" candidate_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f230e9712c42b9a5ab649cace56cb2"
I20260620 18:32:35.509965 13690 raft_consensus.cc:3037] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:35.509996 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35bff5081d4a3db06af0aacf496284" candidate_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165b317e01ed4ff8a0d42b8123f787f7"
I20260620 18:32:35.510219 13556 raft_consensus.cc:3037] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:35.512213 13690 raft_consensus.cc:2454] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab2b178bbfdd4039ae01bc57a5d3d086 in term 1.
I20260620 18:32:35.512486 13556 raft_consensus.cc:2454] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab2b178bbfdd4039ae01bc57a5d3d086 in term 1.
I20260620 18:32:35.512609 13344 leader_election.cc:302] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab2b178bbfdd4039ae01bc57a5d3d086, c9f230e9712c42b9a5ab649cace56cb2; no voters:
I20260620 18:32:35.512930 13769 raft_consensus.cc:2781] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:35.513376 13769 raft_consensus.cc:686] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 1 LEADER]: Becoming Leader. State: Replica: ab2b178bbfdd4039ae01bc57a5d3d086, State: Running, Role: LEADER
I20260620 18:32:35.513895 13769 consensus_queue.cc:237] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } }
I20260620 18:32:35.518275 13259 catalog_manager.cc:5495] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab2b178bbfdd4039ae01bc57a5d3d086 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab2b178bbfdd4039ae01bc57a5d3d086" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 } health_report { overall_health: UNKNOWN } } }
W20260620 18:32:35.522897 13457 tablet.cc:2369] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:35.537979 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.532582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.1:42583)
0620 18:32:35.532762 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.532767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.532838 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:32:35.533126 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.533131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.533220 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.533360 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:35.533731 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.534427 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.535878 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.536255 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.536366 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.536863 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:35.537401 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:35.537553 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.537670 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.537752 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:32:35.545930 13690 raft_consensus.cc:1261] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 1 FOLLOWER]: Refusing update from remote peer ab2b178bbfdd4039ae01bc57a5d3d086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:35.545919 13556 raft_consensus.cc:1261] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 1 FOLLOWER]: Refusing update from remote peer ab2b178bbfdd4039ae01bc57a5d3d086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:35.546867 13774 consensus_queue.cc:1035] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165b317e01ed4ff8a0d42b8123f787f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:35.547235 13769 consensus_queue.cc:1035] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f230e9712c42b9a5ab649cace56cb2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:35.571653 13257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:35.571933 13257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:35.575206 13257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be3b5edcb3354af6ba91169b019579c4: Sending DeleteTablet for 3 replicas of tablet 2d35bff5081d4a3db06af0aacf496284
I20260620 18:32:35.576290 13391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35bff5081d4a3db06af0aacf496284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58268
I20260620 18:32:35.576407 13670 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35bff5081d4a3db06af0aacf496284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53970
I20260620 18:32:35.576426 13536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35bff5081d4a3db06af0aacf496284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41444
I20260620 18:32:35.586555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13296
I20260620 18:32:35.621661 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.587589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.97.1:42583)
0620 18:32:35.587777 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.587785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.590089 (+ 2304us) server_negotiation.cc:407] Connection header received
0620 18:32:35.590176 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.590184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.590302 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.590423 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.592962 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.594613 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.597375 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.598367 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.598520 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.608639 (+ 10119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.608660 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.608687 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.608759 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.614287 (+ 5528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.615569 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.615583 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.615589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.615821 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.617306 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.617314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.617320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.617923 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.618342 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.620705 (+ 2363us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.621078 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:35.622545 13451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:35.625946 13788 tablet_replica.cc:330] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: stopping tablet replica
I20260620 18:32:35.626613 13788 raft_consensus.cc:2229] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:35.627313 13788 raft_consensus.cc:2258] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:35.628643 13789 tablet_replica.cc:330] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: stopping tablet replica
I20260620 18:32:35.631040 13789 raft_consensus.cc:2229] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:35.631167 13788 ts_tablet_manager.cc:1905] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:35.631690 13789 raft_consensus.cc:2258] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:35.633771 13789 ts_tablet_manager.cc:1905] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:35.636562 13788 ts_tablet_manager.cc:1918] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:35.636732 13788 log.cc:1196] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/2d35bff5081d4a3db06af0aacf496284
I20260620 18:32:35.637185 13788 ts_tablet_manager.cc:1939] T 2d35bff5081d4a3db06af0aacf496284 P c9f230e9712c42b9a5ab649cace56cb2: Deleting consensus metadata
I20260620 18:32:35.638937 13244 catalog_manager.cc:4841] TS c9f230e9712c42b9a5ab649cace56cb2 (127.0.97.3:39995): tablet 2d35bff5081d4a3db06af0aacf496284 (table test-table [id=3c4f1614046f46a0a5ba9aa780a80c9e]) successfully deleted
I20260620 18:32:35.640959 13789 ts_tablet_manager.cc:1918] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:35.641119 13789 log.cc:1196] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/2d35bff5081d4a3db06af0aacf496284
I20260620 18:32:35.641673 13789 ts_tablet_manager.cc:1939] T 2d35bff5081d4a3db06af0aacf496284 P 165b317e01ed4ff8a0d42b8123f787f7: Deleting consensus metadata
I20260620 18:32:35.643472 13245 catalog_manager.cc:4841] TS 165b317e01ed4ff8a0d42b8123f787f7 (127.0.97.2:41487): tablet 2d35bff5081d4a3db06af0aacf496284 (table test-table [id=3c4f1614046f46a0a5ba9aa780a80c9e]) successfully deleted
I20260620 18:32:35.730289 13787 tablet_replica.cc:330] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086: stopping tablet replica
I20260620 18:32:35.730823 13787 raft_consensus.cc:2229] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:35.731633 13787 raft_consensus.cc:2258] T 2d35bff5081d4a3db06af0aacf496284 P ab2b178bbfdd4039ae01bc57a5d3d086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:35.732300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13296
W20260620 18:32:35.750029 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42583 (error 108)
I20260620 18:32:35.750923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13460
I20260620 18:32:35.752192 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.751739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58282)
0620 18:32:35.751944 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.752053 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:32:35.752498 13245 catalog_manager.cc:4584] TS ab2b178bbfdd4039ae01bc57a5d3d086 (127.0.97.1:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d35bff5081d4a3db06af0aacf496284: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
I20260620 18:32:35.765041 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.751837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.97.2:41487)
0620 18:32:35.752018 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.752025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.752180 (+ 155us) server_negotiation.cc:407] Connection header received
0620 18:32:35.752371 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.752377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.752457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.752574 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.753838 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.754597 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.755963 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.756354 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.756500 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.759127 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.759146 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.759169 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.759219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.762217 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.762903 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.762912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.762917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.763061 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.763416 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.763422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.763425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.763837 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.764131 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.764452 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.764670 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:35.765754 13596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:35.806427 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.805860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58288)
0620 18:32:35.806102 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.806232 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:32:35.807567 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.807163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58304)
0620 18:32:35.807317 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.807400 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:32:35.860735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13460
I20260620 18:32:35.868399 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.867861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58320)
0620 18:32:35.868157 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.868254 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 18:32:35.869376 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.869039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58326)
0620 18:32:35.869198 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.869271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:32:35.877791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13605
I20260620 18:32:35.892750 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.878659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.0.97.3:39995)
0620 18:32:35.878837 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.878842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.878860 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:35.879081 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.879096 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.879167 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.879265 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.880808 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.881569 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.883045 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.883491 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.883784 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.886299 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.886315 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.886331 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:35.886371 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.889659 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.890509 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.890517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.890519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.890673 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:35.891038 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:35.891043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:35.891045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:35.891453 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.891953 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.892279 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.892481 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:35.893465 13730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:35.950335 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.949761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58336)
0620 18:32:35.950019 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.950134 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 18:32:35.951588 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.951073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42583 (local address 127.0.0.1:58344)
0620 18:32:35.951313 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.951412 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 18:32:35.984104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13605
I20260620 18:32:36.000756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13226
I20260620 18:32:36.002113 13289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:36.116007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13226
I20260620 18:32:36.140039 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:32:36.475250 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:32:36.475497 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:32:36.820634 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:32:36.820 UTC [12277] LOG: received smart shutdown request
2026-06-20 18:32:36.824 UTC [12277] LOG: background worker "logical replication launcher" (PID 12286) exited with exit code 1
2026-06-20 18:32:36.824 UTC [12281] LOG: shutting down
2026-06-20 18:32:37.029 UTC [12277] LOG: database system is shut down
2026-06-20T18:32:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53170 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:37 dist-test-slave-mcnm krb5kdc[13803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:37 dist-test-slave-mcnm krb5kdc[13803](info): set up 2 sockets
Jun 20 18:32:37 dist-test-slave-mcnm krb5kdc[13803](info): commencing operation
krb5kdc: starting...
W20260620 18:32:38.275996 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:32:38 dist-test-slave-mcnm krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:32:38.512228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:38.844694 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:38.845084 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:38.845258 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:38.852669 13816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:38.852855 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:38.852972 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:38.853080 13816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:38.853168 13816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:38.863817 13816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:38.865239 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:38.866602 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:38.878876 13821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:38.878852 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:38.878931 13822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:38.880616 13816 server_base.cc:1034] running on GCE node
I20260620 18:32:38.881461 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:38.883075 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:38.884423 13816 hybrid_clock.cc:648] HybridClock initialized: now 1781980358884359 us; error 116 us; skew 500 ppm
I20260620 18:32:38.884989 13816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:38 dist-test-slave-mcnm krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:38.893872 13816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:38.895768 13816 webserver.cc:460] Webserver started at http://127.0.97.62:34495/ using document root <none> and password file <none>
I20260620 18:32:38.896520 13816 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:38.896632 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:38.896909 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:38.900660 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "085e3b550ecb4266a0ac84853e3f4544"
format_stamp: "Formatted at 2026-06-20 18:32:38 on dist-test-slave-mcnm"
server_key: "5f18d61e22d1473bfe77bb1cf60bb84e"
server_key_iv: "5675d8fa8916f4b005492bbc744a79a4"
server_key_version: "encryptionkey@0"
I20260620 18:32:38.901475 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "085e3b550ecb4266a0ac84853e3f4544"
format_stamp: "Formatted at 2026-06-20 18:32:38 on dist-test-slave-mcnm"
server_key: "5f18d61e22d1473bfe77bb1cf60bb84e"
server_key_iv: "5675d8fa8916f4b005492bbc744a79a4"
server_key_version: "encryptionkey@0"
I20260620 18:32:38.907186 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:38.911074 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:38.912876 13816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:32:38.913230 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "085e3b550ecb4266a0ac84853e3f4544"
format_stamp: "Formatted at 2026-06-20 18:32:38 on dist-test-slave-mcnm"
server_key: "5f18d61e22d1473bfe77bb1cf60bb84e"
server_key_iv: "5675d8fa8916f4b005492bbc744a79a4"
server_key_version: "encryptionkey@0"
I20260620 18:32:38.913441 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:38.932233 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:38.938374 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:38.938776 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:38.960546 13816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44973
I20260620 18:32:38.960578 13882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44973 every 8 connection(s)
I20260620 18:32:38.962077 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:38.965943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13816
I20260620 18:32:38.966181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:32:38.966233 13883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:38.966852 388 external_mini_cluster.cc:1428] Setting key 7532fc3408fb6d11d45d9136dc219264
I20260620 18:32:38.973508 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 18:32:38 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:38.999958 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:38.977199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.97.62:44973)
0620 18:32:38.978101 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:38.978121 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:38.978589 (+ 468us) server_negotiation.cc:407] Connection header received
0620 18:32:38.980747 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:38.980753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:38.981004 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:38.981219 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:38.982695 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:38.984004 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:38.985468 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:38.986286 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:38.986541 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:38.991619 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:38.991630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:38.991651 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:38.991686 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:38.995054 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:38.996075 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:38.996084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:38.996087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:38.996239 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:38.996646 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:38.996651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:38.996661 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:38.997179 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:38.997396 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:38.997929 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 18:32:38.998363 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":636,"thread_start_us":408,"threads_started":1}
I20260620 18:32:38.999938 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:38.977199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44973 (local address 127.0.0.1:51890)
0620 18:32:38.978407 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:38.978448 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:38.980349 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:38.981239 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:38.981246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:38.982113 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:38.982476 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:38.982495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:38.984176 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:38.984183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:38.985312 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:38.985328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:38.986570 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:38.986576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:38.986742 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:38.987462 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:38.987485 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:38.991278 (+ 3793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:38.995421 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:38.995436 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:38.995450 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:38.995936 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:38.996355 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:38.996361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:38.996365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:38.996533 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:38.997329 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:38.997344 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:38.997556 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 18:32:38.997800 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 18:32:38.998465 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":172,"thread_start_us":764,"threads_started":1}
I20260620 18:32:39.003306 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:32:39.004545 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.004081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60948)
0620 18:32:39.004301 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.004413 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 18:32:39.004945 13835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
I20260620 18:32:39.006644 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.006306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60956)
0620 18:32:39.006446 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.006515 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:32:39.006910 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37833: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 18:32:39.019987 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37833
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:39.056313 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } attempt: 1
I20260620 18:32:39.057605 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.057109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60962)
0620 18:32:39.057360 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.057457 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:32:39.058862 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.058406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60972)
0620 18:32:39.058613 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.058691 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 18:32:39.059192 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37833: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
I20260620 18:32:39.127599 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } attempt: 2
I20260620 18:32:39.128808 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.128322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60986)
0620 18:32:39.128584 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.128671 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:32:39.129853 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.129501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:60994)
0620 18:32:39.129645 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.129729 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:32:39.130121 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37833: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
I20260620 18:32:39.221416 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } attempt: 3
I20260620 18:32:39.222561 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.222127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32774)
0620 18:32:39.222333 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.222424 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 18:32:39.223654 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.223263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32778)
0620 18:32:39.223394 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.223466 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:32:39.223883 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37833: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
W20260620 18:32:39.352442 13890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:39.352736 13890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:39.352829 13890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:39.359930 13890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:39.360037 13890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:39.360085 13890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:39.360148 13890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:39.360194 13890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:39.367167 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } attempt: 4
I20260620 18:32:39.368412 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.367895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32784)
0620 18:32:39.368191 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.368280 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 18:32:39.369439 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.369115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32788)
0620 18:32:39.369241 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.369332 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:32:39.369683 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37833: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37833: connect: Connection refused (error 111)
I20260620 18:32:39.370760 13890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:39.372172 13890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:39.373502 13890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:39.385684 13896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:39.385574 13898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:39.386135 13895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:39.387223 13890 server_base.cc:1034] running on GCE node
I20260620 18:32:39.388154 13890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:39.389714 13890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:39.390978 13890 hybrid_clock.cc:648] HybridClock initialized: now 1781980359390928 us; error 66 us; skew 500 ppm
I20260620 18:32:39.391459 13890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:39.394979 13890 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 18:32:39.396638 13890 webserver.cc:460] Webserver started at http://127.0.97.61:41165/ using document root <none> and password file <none>
I20260620 18:32:39.397392 13890 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:39.397542 13890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:39.397859 13890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:39.400978 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data/instance:
uuid: "c7a455ec4cd248bb8f34fe5d098ca421"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "3cf1f35735c9cf6a999263866ad3bc61"
server_key_iv: "3e7149b6b9740d74907cded67f664157"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.401855 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal/instance:
uuid: "c7a455ec4cd248bb8f34fe5d098ca421"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "3cf1f35735c9cf6a999263866ad3bc61"
server_key_iv: "3e7149b6b9740d74907cded67f664157"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.407244 13890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:39.410975 13905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:39.412652 13890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:39.412878 13890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "c7a455ec4cd248bb8f34fe5d098ca421"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "3cf1f35735c9cf6a999263866ad3bc61"
server_key_iv: "3e7149b6b9740d74907cded67f664157"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.413120 13890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:39.434713 13890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:39.439004 13890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:39.439316 13890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:39.458655 13890 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37833
I20260620 18:32:39.458693 13956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37833 every 8 connection(s)
I20260620 18:32:39.460139 13890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
I20260620 18:32:39.462982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13890
I20260620 18:32:39.463162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-1/wal/instance
I20260620 18:32:39.463740 388 external_mini_cluster.cc:1428] Setting key 16dbd97d1fe3e540b3b849ac40f9964b
I20260620 18:32:39.464354 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:39.473742 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:39.497846 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.476775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.97.62:44973)
0620 18:32:39.476962 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.476967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.478357 (+ 1390us) server_negotiation.cc:407] Connection header received
0620 18:32:39.480337 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.480343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.480437 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.480551 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.482158 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.482845 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.484485 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.484916 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.485055 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.489968 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.489978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.489987 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.490038 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:39.494103 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.495024 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.495033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.495040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.495198 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.495807 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.495815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.495821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.496250 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:39.496383 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:39.497403 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0620 18:32:39.497605 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:32:39.499382 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.477335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44973 (local address 127.0.0.1:51892)
0620 18:32:39.478171 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.478219 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:39.480054 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:39.480802 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:39.480811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:39.481352 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:39.481968 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.481988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.483018 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.483022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.484335 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.484347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.485145 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.485152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.485385 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.486125 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:39.486148 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:39.489680 (+ 3532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:39.494324 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.494332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.494348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.494868 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.495342 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.495347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.495349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.495636 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.496406 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:39.496415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:39.496938 (+ 523us) client_negotiation.cc:769] Sending connection context
0620 18:32:39.497290 (+ 352us) client_negotiation.cc:240] Negotiation successful
0620 18:32:39.497882 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":379,"threads_started":1}
I20260620 18:32:39.501336 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:32:39.517628 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.502011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32792)
0620 18:32:39.502331 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.502352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:39.502463 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:39.503399 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:39.503407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:39.504001 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:39.504216 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.504240 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.505578 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.505587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.507157 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.507176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.508124 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.508137 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.508404 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.509039 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:39.509069 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:39.511629 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:39.514845 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.514852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.514856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.515274 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.515719 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.515724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.515728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.515862 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.516483 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:39.516488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:39.516661 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 18:32:39.516831 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:32:39.517205 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 18:32:39.517783 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.502230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.0.97.61:37833)
0620 18:32:39.502962 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.502968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.502994 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:39.503038 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.503044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.503227 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.503344 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.504382 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.505408 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.507319 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.507943 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.508042 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.511844 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.511855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.511869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.511901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:39.514709 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.515405 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.515414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.515417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.515608 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.515964 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.515972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.515975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.516355 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:39.516571 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:39.516969 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:32:39.517369 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":581,"thread_start_us":407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 18:32:39.520606 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:32:39.521174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33559
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:39.521967 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.521213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33618)
0620 18:32:39.521582 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.521723 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
W20260620 18:32:39.522240 13909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.523972 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.523492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33622)
0620 18:32:39.523703 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.523818 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 18:32:39.524190 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.573544 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 1
I20260620 18:32:39.574729 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.574290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33630)
0620 18:32:39.574485 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.574585 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:32:39.575794 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.575419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33638)
0620 18:32:39.575607 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.575684 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:32:39.576009 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.644235 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 2
I20260620 18:32:39.645550 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.644977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33646)
0620 18:32:39.645277 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.645387 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 18:32:39.646590 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.646231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33648)
0620 18:32:39.646384 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.646463 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:32:39.646812 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.668948 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } attempt: 5
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:32:39.683508 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.669906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.97.61:37833)
0620 18:32:39.670126 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.670133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.670157 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:39.670222 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.670227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.670304 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.670413 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.671258 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.672030 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.673189 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.673684 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.673826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.677973 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.677978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.677982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.678016 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:39.680946 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.681608 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.681615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.681619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.681758 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.682137 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.682147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.682149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.682430 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:39.682581 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:39.682968 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:32:39.683189 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:32:39.683516 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.669720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32798)
0620 18:32:39.669917 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.669943 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:39.670098 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:39.670455 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:39.670460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:39.670890 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:39.671065 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.671074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.672195 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.672199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.673047 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.673057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.673927 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.673935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.674202 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.675023 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:39.675053 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:39.677761 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:39.681099 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.681105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.681109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.681488 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.681873 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.681878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.681882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.682019 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.682547 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:39.682552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:39.682717 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:32:39.682863 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:32:39.683153 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:32:39.684710 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:32:39.685706 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.685327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33660)
0620 18:32:39.685487 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.685566 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:32:39.686580 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.686275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33676)
0620 18:32:39.686412 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.686490 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:32:39.686779 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.738081 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 3
I20260620 18:32:39.738059 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 1
I20260620 18:32:39.739414 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.738853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33696)
0620 18:32:39.739132 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.739228 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:32:39.739842 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.738888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33680)
0620 18:32:39.739227 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.739350 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 18:32:39.740782 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.740347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33698)
0620 18:32:39.740529 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.740607 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:32:39.741004 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.741300 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.740612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33708)
0620 18:32:39.741040 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.741160 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260620 18:32:39.741537 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.809279 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 2
I20260620 18:32:39.810475 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.809973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33710)
0620 18:32:39.810210 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.810327 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:32:39.811462 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.811146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33714)
0620 18:32:39.811284 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.811358 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:32:39.811776 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
W20260620 18:32:39.853432 13964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:39.853773 13964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:39.853935 13964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:39.861125 13964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:39.861279 13964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:39.861395 13964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:39.861493 13964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:39.861573 13964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:39.871758 13964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44973,127.0.97.61:37833,127.0.97.60:33559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:39.873154 13964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:39.874542 13964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:32:39.884867 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 4
I20260620 18:32:39.886114 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.885534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33726)
0620 18:32:39.885781 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.885896 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 18:32:39.887045 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:39.887004 13970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:39.887244 13969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:39.887951 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.887458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33736)
0620 18:32:39.887703 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.887797 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:32:39.887928 13964 server_base.cc:1034] running on GCE node
W20260620 18:32:39.888337 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.888916 13964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:39.890300 13964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:39.891551 13964 hybrid_clock.cc:648] HybridClock initialized: now 1781980359891461 us; error 35 us; skew 500 ppm
I20260620 18:32:39.891943 13964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:39.895162 13964 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 18:32:39.896607 13964 webserver.cc:460] Webserver started at http://127.0.97.60:37365/ using document root <none> and password file <none>
I20260620 18:32:39.897330 13964 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:39.897432 13964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:39.897670 13964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:39.900767 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data/instance:
uuid: "3f3df928c36f4d7686b147abf710d8dd"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "62f01222d13d33ea74faddd11e0d0f8e"
server_key_iv: "9466fcc110314ad1269de97fe60cac52"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.901528 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal/instance:
uuid: "3f3df928c36f4d7686b147abf710d8dd"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "62f01222d13d33ea74faddd11e0d0f8e"
server_key_iv: "9466fcc110314ad1269de97fe60cac52"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.906956 13964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:39.910523 13979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:39.911999 13964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:39.912185 13964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "3f3df928c36f4d7686b147abf710d8dd"
format_stamp: "Formatted at 2026-06-20 18:32:39 on dist-test-slave-mcnm"
server_key: "62f01222d13d33ea74faddd11e0d0f8e"
server_key_iv: "9466fcc110314ad1269de97fe60cac52"
server_key_version: "encryptionkey@0"
I20260620 18:32:39.912391 13964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:39.917989 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 3
I20260620 18:32:39.919185 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.918681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33752)
0620 18:32:39.918909 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.919003 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:32:39.920271 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.919910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33754)
0620 18:32:39.920059 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.920139 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 18:32:39.920470 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33559: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33559: connect: Connection refused (error 111)
I20260620 18:32:39.928668 13964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:39.933477 13964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:39.933766 13964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:39.952931 13964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33559
I20260620 18:32:39.952981 14030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33559 every 8 connection(s)
I20260620 18:32:39.954432 13964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
I20260620 18:32:39.958184 14031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:39.964262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 13964
I20260620 18:32:39.964468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980152255989-388-0/minicluster-data/master-2/wal/instance
I20260620 18:32:39.965015 388 external_mini_cluster.cc:1428] Setting key 48da3808fb1719c05ed0f7fb342725a4
I20260620 18:32:39.965039 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:32:39.986620 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.968602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.0.97.62:44973)
0620 18:32:39.968875 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.968884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.969611 (+ 727us) server_negotiation.cc:407] Connection header received
0620 18:32:39.971707 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.971712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.971787 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.971903 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.973470 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.974170 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.975838 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.976266 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.976405 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.980263 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.980269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.980272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.980307 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:39.983358 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.984283 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.984289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.984293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.984430 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.984843 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.984852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.984855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.985110 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:39.985298 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:39.986016 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 18:32:39.986262 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:39.987800 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.966888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.0.97.60:33559)
0620 18:32:39.969761 (+ 2873us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.969767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.969804 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:32:39.971383 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.971404 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.971739 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.971934 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.973305 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.974642 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.976216 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.976746 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.976886 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.981019 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.981028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.981040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.981104 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:39.984137 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.984831 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.984837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.984840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.984957 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:39.985414 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:39.985425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:39.985427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:39.985776 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:39.986106 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:39.986428 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:32:39.986676 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2685,"thread_start_us":2482,"threads_started":1}
I20260620 18:32:39.988101 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.968217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44973 (local address 127.0.0.1:51908)
0620 18:32:39.969460 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.969510 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:39.971411 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:39.972368 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:39.972372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:39.972896 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:39.973292 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.973307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.974350 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.974356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.975694 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.975706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.976520 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.976529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.976742 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.977485 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:39.977519 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:39.979997 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:39.983555 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.983567 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.983580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.984150 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.984541 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:39.984547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:39.984551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:39.984724 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:39.985290 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:39.985304 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:39.985652 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 18:32:39.985919 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:32:39.986411 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":865,"threads_started":1}
I20260620 18:32:39.989528 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 18:32:39 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:40.003818 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.990189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37833 (local address 127.0.0.1:32804)
0620 18:32:39.990457 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:39.990476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:39.990585 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:39.990981 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:39.990985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:39.991362 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:39.991553 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.991568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.992562 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.992567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.993692 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:39.993705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.994446 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.994452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:39.994672 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.995302 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:39.995320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:39.997675 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:40.001087 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.001093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.001098 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.001489 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.001887 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.001892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.001896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.002045 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.002756 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:40.002761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:40.002931 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:32:40.003101 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:32:40.003402 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 18:32:40.003858 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.990336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.0.97.61:37833)
0620 18:32:39.990541 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.990547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.990567 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:39.990727 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.990731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.990836 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.990962 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.991717 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.992418 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.993869 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.994290 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.994435 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.997904 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.997908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.997912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.997948 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.000929 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.001617 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.001624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.001635 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.001773 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.002162 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.002167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.002177 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.002511 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.002692 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.003234 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.003442 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:40.005165 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:32:40.005823 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:39.990953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.97.62:44973)
0620 18:32:39.991115 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:39.991120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:39.991250 (+ 130us) server_negotiation.cc:407] Connection header received
0620 18:32:39.991426 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:39.991431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:39.991501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:39.991640 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:39.992853 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.993593 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.994918 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:39.995361 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:39.995498 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:39.999865 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:39.999870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:39.999873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:39.999906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.002662 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.003675 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.003683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.003687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.003847 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.004278 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.004283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.004286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.004648 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.004923 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.005286 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.005503 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:32:40 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 18:32:40 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:32:40.020383 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.005898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33768)
0620 18:32:40.006120 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:40.006150 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:40.006244 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:40.006541 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:40.006545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:40.006952 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:40.007080 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.007088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.008234 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.008239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.009697 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.009713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.010752 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.010756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.010902 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.011452 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:40.011476 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:40.013866 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:40.017720 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.017726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.017730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.018202 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.018672 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.018677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.018682 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.018826 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.019359 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:40.019364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:40.019504 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 18:32:40.019797 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 18:32:40.020088 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":4}
I20260620 18:32:40.020704 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.006120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.97.60:33559)
0620 18:32:40.006271 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.006276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.006293 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:32:40.006363 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.006366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.006414 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.006483 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.007234 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.008065 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.009948 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.010609 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.010748 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.014080 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.014085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.014088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.014144 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.017553 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.018323 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.018330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.018335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.018476 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.018938 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.018945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.018948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.019245 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.019396 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.019930 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.020317 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":51}
I20260620 18:32:40.025367 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.007802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.97.61:37833)
0620 18:32:40.007969 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.007974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.007991 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:32:40.008036 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.008040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.008108 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.008203 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.009924 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.010956 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.012783 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.013205 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.013368 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.018940 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.018946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.018952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.019044 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.022233 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.023265 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.023270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.023272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.023401 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.023892 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.023897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.023899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.024204 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.024410 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.024810 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.024982 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:32:40.026791 14031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: Bootstrap starting.
I20260620 18:32:40.030390 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:40.030547 14031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:40.031663 14031 log.cc:824] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:40.034657 14031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: No bootstrap required, opened a new log
I20260620 18:32:40.045663 14031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.046131 14031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:40.046253 14031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3df928c36f4d7686b147abf710d8dd, State: Initialized, Role: FOLLOWER
I20260620 18:32:40.047091 14031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.048969 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.049414 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.050396 14031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:40.056926 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.035734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.97.62:44973)
0620 18:32:40.035935 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.035942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.036390 (+ 448us) server_negotiation.cc:407] Connection header received
0620 18:32:40.036684 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.036693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.036798 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.036903 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.039685 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.041142 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.044059 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.044734 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.044823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.049017 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.049025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.049031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.049072 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.053371 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.054323 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.054331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.054335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.054507 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.054957 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.054961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.054964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.055353 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.055626 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.056277 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.056519 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:40.058386 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.036563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.97.61:37833)
0620 18:32:40.036748 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.036756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.037315 (+ 559us) server_negotiation.cc:407] Connection header received
0620 18:32:40.037568 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.037573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.037656 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.037731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.039999 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.041107 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.043349 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.044087 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.044183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.049742 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.049748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.049753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.049801 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.053752 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.055065 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.055074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.055080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.055275 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.055851 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.055859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.055865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.056334 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.056609 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.057674 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.057968 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:40.062592 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.037321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.97.60:33559)
0620 18:32:40.037512 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.037519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.044325 (+ 6806us) server_negotiation.cc:407] Connection header received
0620 18:32:40.044738 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.044744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.044822 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.044886 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.047379 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.048575 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.050510 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.051394 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.051547 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.054988 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.054996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.055000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.055041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.058446 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.059803 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.059812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.059818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.060067 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.060642 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.060650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.060656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.061116 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.061378 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.061796 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.062099 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:32:40.071244 14055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:32:40.071389 14055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:32:40.066833 13964 master_runner.cc:186] Error getting master registration for 127.0.97.62:44973: OK, instance_id { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" instance_seqno: 1781980358954398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:32:40.076102 13964 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:32:40.097688 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 4
Jun 20 18:32:40 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:32:40.114346 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.098753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.0.97.60:33559)
0620 18:32:40.098970 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.098982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.099003 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:32:40.099067 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.099072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.099151 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.099262 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.100056 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.101347 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.103175 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.103759 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.103862 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.107941 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.107946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.107950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.107982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.111670 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.112428 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.112432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.112434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.112582 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.112984 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.112987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.112989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.113277 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.113386 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.113812 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.113982 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:32:40.114431 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.098462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33790)
0620 18:32:40.098696 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:40.098718 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:40.098846 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:40.099290 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:40.099294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:40.099738 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:40.099901 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.099910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.101520 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.101525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.102990 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.103019 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.103919 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.103926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.104126 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.104813 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:40.104839 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:40.107721 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:40.111793 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.111799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.111803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.112309 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.112695 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.112701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.112706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.112880 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.113393 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:40.113398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:40.113556 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:32:40.113737 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 18:32:40.114060 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:32:40.119472 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: Bootstrap starting.
I20260620 18:32:40.122989 13883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:40.124280 13883 log.cc:824] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:40.126441 14044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:40.126747 14044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.127254 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: No bootstrap required, opened a new log
I20260620 18:32:40.128440 14044 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085e3b550ecb4266a0ac84853e3f4544 (127.0.97.62:44973), c7a455ec4cd248bb8f34fe5d098ca421 (127.0.97.61:37833)
I20260620 18:32:40.139405 13883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.139861 13883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:40.139986 13883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085e3b550ecb4266a0ac84853e3f4544, State: Initialized, Role: FOLLOWER
I20260620 18:32:40.140630 13883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.141607 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.142019 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.142835 13883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:40.142932 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3df928c36f4d7686b147abf710d8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085e3b550ecb4266a0ac84853e3f4544" is_pre_election: true
I20260620 18:32:40.143582 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3df928c36f4d7686b147abf710d8dd in term 0.
I20260620 18:32:40.144345 13983 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085e3b550ecb4266a0ac84853e3f4544, 3f3df928c36f4d7686b147abf710d8dd; no voters:
I20260620 18:32:40.144881 14044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:40.145035 14044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:40.145131 14044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:40.148479 14044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.149701 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3df928c36f4d7686b147abf710d8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085e3b550ecb4266a0ac84853e3f4544"
I20260620 18:32:40.149922 13858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:40.150290 14044 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [CANDIDATE]: Term 1 election: Requested vote from peers 085e3b550ecb4266a0ac84853e3f4544 (127.0.97.62:44973), c7a455ec4cd248bb8f34fe5d098ca421 (127.0.97.61:37833)
I20260620 18:32:40.153329 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3df928c36f4d7686b147abf710d8dd in term 1.
I20260620 18:32:40.153921 13983 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085e3b550ecb4266a0ac84853e3f4544, 3f3df928c36f4d7686b147abf710d8dd; no voters:
I20260620 18:32:40.154223 14044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:40.161494 14044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [term 1 LEADER]: Becoming Leader. State: Replica: 3f3df928c36f4d7686b147abf710d8dd, State: Running, Role: LEADER
I20260620 18:32:40.162261 14044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.167155 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3df928c36f4d7686b147abf710d8dd. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.167876 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:40.171499 14067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:40.174078 14067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 18:32:40.185860 13985 tablet.cc:2369] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:40.187584 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } attempt: 5
W20260620 18:32:40.195951 13837 tablet.cc:2369] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:40.199488 14072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:32:40.199637 14072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 18:32:40 dist-test-slave-mcnm krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:32:40.177541 13816 master_runner.cc:186] Error getting master registration for 127.0.97.61:37833: OK, instance_id { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" instance_seqno: 1781980359453096 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:32:40.205631 13816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:32:40.221297 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.188353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33559 (local address 127.0.0.1:33798)
0620 18:32:40.188849 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:40.188876 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:40.189023 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:40.190309 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:40.190317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:40.191076 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:40.191361 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.191377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.193703 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.193712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.195712 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:40.195733 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.198456 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.198463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:40.198874 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.199941 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:40.199996 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:40.204677 (+ 4681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:40.214766 (+ 10089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.214799 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.214804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.215948 (+ 1144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.216841 (+ 893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:40.216848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:40.216881 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:40.217304 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:40.218496 (+ 1192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:40.218501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:40.218790 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 18:32:40.218991 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 18:32:40.219995 (+ 1004us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
I20260620 18:32:40.222412 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:40.189472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.97.60:33559)
0620 18:32:40.189719 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:40.189727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:40.189751 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:40.189832 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:40.189840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:40.189941 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:40.190062 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:40.191597 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.193190 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.197215 (+ 4025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:40.198244 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:40.198408 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:40.204999 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:40.205007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:40.205013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:40.205056 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:40.211814 (+ 6758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.216134 (+ 4320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.216140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.216207 (+ 67us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.216605 (+ 398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:40.217451 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:40.217482 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:40.217486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:40.218298 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:40.219404 (+ 1106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:40.219938 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 18:32:40.220868 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:32:40.227558 13858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [term 1 FOLLOWER]: Refusing update from remote peer 3f3df928c36f4d7686b147abf710d8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:40.228648 14065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:40.236720 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: Bootstrap starting.
I20260620 18:32:40.240914 13957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:40.242213 13957 log.cc:824] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:40.245543 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3df928c36f4d7686b147abf710d8dd. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.245889 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.248118 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: No bootstrap required, opened a new log
I20260620 18:32:40.253691 14074 mvcc.cc:205] Tried to move back new op lower bound from 7298991555477704704 to 7298991555230007296. Current Snapshot: MvccSnapshot[applied={T|T < 7298991555477704704}]
I20260620 18:32:40.254222 14075 mvcc.cc:205] Tried to move back new op lower bound from 7298991555477704704 to 7298991555230007296. Current Snapshot: MvccSnapshot[applied={T|T < 7298991555477704704}]
I20260620 18:32:40.270015 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.270468 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:40.271093 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.271366 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.275727 13957 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.276145 13957 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:40.276281 13957 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a455ec4cd248bb8f34fe5d098ca421, State: Initialized, Role: FOLLOWER
I20260620 18:32:40.276909 14067 catalog_manager.cc:1347] Generated new cluster ID: 0738c5e687a14d029a82056d419598f3
I20260620 18:32:40.277015 14067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:40.277093 13957 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } }
I20260620 18:32:40.292094 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.292656 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.293622 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3df928c36f4d7686b147abf710d8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" is_pre_election: true
I20260620 18:32:40.294348 13930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:40.295161 13957 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:40.298617 13930 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 1 FOLLOWER]: Refusing update from remote peer 3f3df928c36f4d7686b147abf710d8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:40.299194 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3df928c36f4d7686b147abf710d8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a455ec4cd248bb8f34fe5d098ca421"
I20260620 18:32:40.299651 14065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:40.302788 13931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3df928c36f4d7686b147abf710d8dd in term 1.
I20260620 18:32:40.349032 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3df928c36f4d7686b147abf710d8dd. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.349399 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.349836 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.350222 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:40.356221 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f3df928c36f4d7686b147abf710d8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085e3b550ecb4266a0ac84853e3f4544" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44973 } } peers { permanent_uuid: "c7a455ec4cd248bb8f34fe5d098ca421" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37833 } } peers { permanent_uuid: "3f3df928c36f4d7686b147abf710d8dd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33559 } } }
I20260620 18:32:40.356606 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:40.366499 14067 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:40.369217 14067 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:40.398233 14097 catalog_manager.cc:1259] Loaded cluster ID: 0738c5e687a14d029a82056d419598f3
I20260620 18:32:40.398366 14097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: loading cluster ID for follower catalog manager: success
I20260620 18:32:40.401815 14097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: acquiring CA information for follower catalog manager: success
W20260620 18:32:40.403026 14097 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 18:32:40.407469 14067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3df928c36f4d7686b147abf710d8dd: Generated new TSK 0
I20260620 18:32:40.408492 14067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:41.201614 14072 catalog_manager.cc:1259] Loaded cluster ID: 0738c5e687a14d029a82056d419598f3
I20260620 18:32:41.201767 14072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: loading cluster ID for follower catalog manager: success
I20260620 18:32:41.204375 14072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: acquiring CA information for follower catalog manager: success
I20260620 18:32:41.206144 14072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 085e3b550ecb4266a0ac84853e3f4544: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:32:41.405963 14097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7a455ec4cd248bb8f34fe5d098ca421: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:32:41.554498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13816
I20260620 18:32:41.555778 13878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:41.669556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13816
I20260620 18:32:41.690901 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13890
I20260620 18:32:41.692065 13952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:41.800920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13890
I20260620 18:32:41.822257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 13964
I20260620 18:32:41.823436 14026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:41.932512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 13964
2026-06-20T18:32:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4761 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:42 dist-test-slave-mcnm krb5kdc[14109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:42 dist-test-slave-mcnm krb5kdc[14109](info): set up 2 sockets
Jun 20 18:32:42 dist-test-slave-mcnm krb5kdc[14109](info): commencing operation
krb5kdc: starting...
W20260620 18:32:43.056116 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:32:43 dist-test-slave-mcnm krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:32:43.292630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32997
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:43.617650 14122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:43.618000 14122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:43.618153 14122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:43.625326 14122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:43.625483 14122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:43.625589 14122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:43.625691 14122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:43.625771 14122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:43.635812 14122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:43.637148 14122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:43.638506 14122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:43.650964 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:43.651052 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:43.651017 14130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:43.651873 14122 server_base.cc:1034] running on GCE node
I20260620 18:32:43.652767 14122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:43.654422 14122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:43.655640 14122 hybrid_clock.cc:648] HybridClock initialized: now 1781980363655595 us; error 53 us; skew 500 ppm
I20260620 18:32:43.656049 14122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:43 dist-test-slave-mcnm krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:43.659400 14122 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:43.661068 14122 webserver.cc:460] Webserver started at http://127.0.97.62:35575/ using document root <none> and password file <none>
I20260620 18:32:43.661785 14122 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:43.661903 14122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:43.662187 14122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:43.665473 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "14d257e634c842aabaaba2b8cc85892b"
format_stamp: "Formatted at 2026-06-20 18:32:43 on dist-test-slave-mcnm"
server_key: "3601b7a8544e085cfb9ef06f63026248"
server_key_iv: "a1b828017e0f5b737f77536a6bc7a2db"
server_key_version: "encryptionkey@0"
I20260620 18:32:43.666261 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "14d257e634c842aabaaba2b8cc85892b"
format_stamp: "Formatted at 2026-06-20 18:32:43 on dist-test-slave-mcnm"
server_key: "3601b7a8544e085cfb9ef06f63026248"
server_key_iv: "a1b828017e0f5b737f77536a6bc7a2db"
server_key_version: "encryptionkey@0"
I20260620 18:32:43.672021 14122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 18:32:43.675966 14137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:43.677476 14122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:43.677637 14122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "14d257e634c842aabaaba2b8cc85892b"
format_stamp: "Formatted at 2026-06-20 18:32:43 on dist-test-slave-mcnm"
server_key: "3601b7a8544e085cfb9ef06f63026248"
server_key_iv: "a1b828017e0f5b737f77536a6bc7a2db"
server_key_version: "encryptionkey@0"
I20260620 18:32:43.677819 14122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:43.700044 14122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:43.704456 14122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:43.704720 14122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:43.724083 14122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32997
I20260620 18:32:43.724162 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32997 every 8 connection(s)
I20260620 18:32:43.725504 14122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:43.729360 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:43.735026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14122
I20260620 18:32:43.735287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:32:43.736008 388 external_mini_cluster.cc:1428] Setting key 1c2b9d827e642276d1b4da4549284862
I20260620 18:32:43.740238 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b: Bootstrap starting.
I20260620 18:32:43.744110 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:43.745308 14189 log.cc:824] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:43.748735 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b: No bootstrap required, opened a new log
Jun 20 18:32:43 dist-test-slave-mcnm krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:43.759459 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260620 18:32:43.759961 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:43.760062 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d257e634c842aabaaba2b8cc85892b, State: Initialized, Role: FOLLOWER
I20260620 18:32:43.760722 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260620 18:32:43.761013 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:43.761158 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:43.761332 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:43.737871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.97.62:32997)
0620 18:32:43.738703 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.738715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.738755 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:32:43.740532 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.740556 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.740946 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.741264 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.743103 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.744613 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.746446 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.747193 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.747354 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:43.752454 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:43.752465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:43.752485 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:43.752537 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:43.756634 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.757374 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.757389 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.757398 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.757624 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.758018 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.758023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.758028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.758752 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:43.759051 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:43.759449 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:32:43.759927 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":406,"threads_started":1}
I20260620 18:32:43.761371 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:43.764372 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260620 18:32:43.764962 14189 leader_election.cc:302] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d257e634c842aabaaba2b8cc85892b; no voters:
I20260620 18:32:43.765667 14189 leader_election.cc:288] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:43.765763 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:43.765880 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:43.767676 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:43.769791 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [term 1 LEADER]: Becoming Leader. State: Replica: 14d257e634c842aabaaba2b8cc85892b, State: Running, Role: LEADER
I20260620 18:32:43.770326 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260620 18:32:43.773417 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d257e634c842aabaaba2b8cc85892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } } }
I20260620 18:32:43.773470 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d257e634c842aabaaba2b8cc85892b. Latest consensus state: current_term: 1 leader_uuid: "14d257e634c842aabaaba2b8cc85892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d257e634c842aabaaba2b8cc85892b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } } }
I20260620 18:32:43.773815 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:43.773837 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:43.774816 14204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:43.780885 14204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:43.790405 14204 catalog_manager.cc:1347] Generated new cluster ID: 9048e6c31b5b4cf8a909e36b4307ff99
I20260620 18:32:43.790563 14204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:43.811951 14204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:43.814100 14204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:43.831182 14204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d257e634c842aabaaba2b8cc85892b: Generated new TSK 0
I20260620 18:32:43.832240 14204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:43.880347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14122
I20260620 18:32:43.881644 14184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:43.977978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14122
2026-06-20T18:32:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2046 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:44 dist-test-slave-mcnm krb5kdc[14217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:44 dist-test-slave-mcnm krb5kdc[14217](info): set up 2 sockets
Jun 20 18:32:44 dist-test-slave-mcnm krb5kdc[14217](info): commencing operation
krb5kdc: starting...
W20260620 18:32:45.088333 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:32:45 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:32:45.324512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:45.645193 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:45.645470 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:45.645646 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:45.652819 14230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:45.652925 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:45.653019 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:45.653101 14230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:45.653165 14230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:45.662703 14230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:45.664011 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:45.665294 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:45.677088 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.677119 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.677119 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:45.678236 14230 server_base.cc:1034] running on GCE node
I20260620 18:32:45.678977 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:45.680421 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:45.681679 14230 hybrid_clock.cc:648] HybridClock initialized: now 1781980365681623 us; error 37 us; skew 500 ppm
I20260620 18:32:45.682022 14230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:45 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:45.685322 14230 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:45.687096 14230 webserver.cc:460] Webserver started at http://127.0.97.62:33311/ using document root <none> and password file <none>
I20260620 18:32:45.687902 14230 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:45.688047 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:45.688364 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:45.691372 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "57b3357baad2443ba15f692da054b775"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-mcnm"
server_key: "08f002f5a033b3cb0c42163e462bf25e"
server_key_iv: "e8e22c2f3d922f3936042c6865cdf055"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.692198 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "57b3357baad2443ba15f692da054b775"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-mcnm"
server_key: "08f002f5a033b3cb0c42163e462bf25e"
server_key_iv: "e8e22c2f3d922f3936042c6865cdf055"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.697153 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:45.700703 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:45.702142 14230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:32:45.702335 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "57b3357baad2443ba15f692da054b775"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-mcnm"
server_key: "08f002f5a033b3cb0c42163e462bf25e"
server_key_iv: "e8e22c2f3d922f3936042c6865cdf055"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.702550 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:45.734153 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:45.738445 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:45.738758 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:45.758177 14230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34605
I20260620 18:32:45.758208 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34605 every 8 connection(s)
I20260620 18:32:45.759630 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:45.763602 14297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:45.766181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14230
I20260620 18:32:45.766395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:32:45.767222 388 external_mini_cluster.cc:1428] Setting key 22da28df8a1999e126683c146c01d874
I20260620 18:32:45.775449 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap starting.
I20260620 18:32:45.778481 14297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:45.779614 14297 log.cc:824] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:32:45 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:45.782848 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: No bootstrap required, opened a new log
I20260620 18:32:45.790705 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:45.769596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.97.62:34605)
0620 18:32:45.770404 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:45.770416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:45.770455 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:32:45.772234 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:45.772258 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:45.772670 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:45.773100 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:45.774532 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.775931 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.777252 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.777911 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.778103 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:45.782863 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:45.782874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:45.782889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:45.782965 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:45.786361 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.787101 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.787110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.787120 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.787320 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.787779 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.787784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.787789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.788380 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:45.788713 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:45.789163 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 18:32:45.789617 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":378,"threads_started":1}
I20260620 18:32:45.791370 14297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:45.791803 14297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:45.791919 14297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b3357baad2443ba15f692da054b775, State: Initialized, Role: FOLLOWER
I20260620 18:32:45.792639 14297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:45.792918 14297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:45.793058 14297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:45.793253 14297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:45.795748 14297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:45.796176 14297 leader_election.cc:302] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b3357baad2443ba15f692da054b775; no voters:
I20260620 18:32:45.796761 14297 leader_election.cc:288] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:45.796962 14302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:45.797742 14302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 LEADER]: Becoming Leader. State: Replica: 57b3357baad2443ba15f692da054b775, State: Running, Role: LEADER
I20260620 18:32:45.798280 14302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:45.798985 14297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:45.801257 14303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:45.801229 14304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b3357baad2443ba15f692da054b775. Latest consensus state: current_term: 1 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:45.801712 14304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:45.801712 14303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:45.802866 14307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:45.810730 14307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:45.821789 14307 catalog_manager.cc:1347] Generated new cluster ID: 65f0dae2796c4eda9b56ab4de0f5dec9
I20260620 18:32:45.821938 14307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:45.842121 14307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:45.843474 14307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:45.854167 14307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Generated new TSK 0
I20260620 18:32:45.855048 14307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:32:45.923395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:46.265021 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:46.265377 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:46.265594 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:46.272938 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:46.273123 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:46.273334 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:32:46.283257 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:46.284679 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:46.286052 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:46.300005 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.299958 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.300042 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:46.301427 14323 server_base.cc:1034] running on GCE node
I20260620 18:32:46.302007 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:46.302963 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:46.304217 14323 hybrid_clock.cc:648] HybridClock initialized: now 1781980366304173 us; error 51 us; skew 500 ppm
I20260620 18:32:46.304562 14323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:46 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:46.308074 14323 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:32:46.309594 14323 webserver.cc:460] Webserver started at http://127.0.97.1:34749/ using document root <none> and password file <none>
I20260620 18:32:46.310302 14323 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:46.310454 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:46.310763 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:46.313714 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "29d9634705a8442ea9dcd70a0e4245e1"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "76f0ed337946d42644af6bbedb619396"
server_key_iv: "8897464de3a7e0099766614c8710e77d"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.314512 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29d9634705a8442ea9dcd70a0e4245e1"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "76f0ed337946d42644af6bbedb619396"
server_key_iv: "8897464de3a7e0099766614c8710e77d"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.319878 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:32:46.323302 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.324644 14323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:46.324826 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "29d9634705a8442ea9dcd70a0e4245e1"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "76f0ed337946d42644af6bbedb619396"
server_key_iv: "8897464de3a7e0099766614c8710e77d"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.325047 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:46.346951 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:46.351034 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:46.351330 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:46.352309 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:46.353807 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:46.353920 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.354063 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:46.354133 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.388419 14323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43715
I20260620 18:32:46.388510 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43715 every 8 connection(s)
I20260620 18:32:46.389983 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:46.399737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14323
I20260620 18:32:46.399969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:46.400732 388 external_mini_cluster.cc:1428] Setting key 5cdac719536cfe0c6e854194f14bb9bc
Jun 20 18:32:46 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:46.411059 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.392731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37667 (local address 127.0.97.62:34605)
0620 18:32:46.393243 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:46.393248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:46.393982 (+ 734us) server_negotiation.cc:407] Connection header received
0620 18:32:46.395887 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:46.395893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:46.395983 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:46.396110 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:46.397586 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.398335 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.400108 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.400577 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.400712 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.404334 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:46.404340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:46.404343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:46.404376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:46.407647 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.408506 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.408512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.408523 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.408658 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.409158 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.409162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.409165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.409442 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:46.409628 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:46.410410 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 18:32:46.410666 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
I20260620 18:32:46.412213 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.393107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:37667)
0620 18:32:46.393817 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:46.393856 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:46.395617 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:46.396354 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:46.396362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:46.396870 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:46.397411 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.397427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.398483 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.398495 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.399951 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.399963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.400818 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.400825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.401059 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.401737 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:46.401766 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:46.404105 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:46.407807 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.407818 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.407837 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.408357 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.408782 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.408788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.408792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.409007 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.409584 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:46.409603 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:46.409988 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 18:32:46.410280 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 18:32:46.410869 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260620 18:32:46.413944 14451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:46.414265 14451 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:46.415021 14451 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:46.417506 14262 ts_manager.cc:194] Registered new tserver with Master: 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715)
I20260620 18:32:46.419829 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37667
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:32:46.454905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:46.782812 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:46.783072 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:46.783227 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:46.790649 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:46.790762 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:46.790933 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:32:46.801002 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:46.802238 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:46.803555 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:46.816833 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.817057 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.816850 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:46.817863 14457 server_base.cc:1034] running on GCE node
I20260620 18:32:46.818435 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:46.819335 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:46.820585 14457 hybrid_clock.cc:648] HybridClock initialized: now 1781980366820528 us; error 69 us; skew 500 ppm
I20260620 18:32:46.820977 14457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:46 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:46.824496 14457 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:32:46.826581 14457 webserver.cc:460] Webserver started at http://127.0.97.2:46167/ using document root <none> and password file <none>
I20260620 18:32:46.827298 14457 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:46.827441 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:46.827786 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:46.830972 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "1b9b818b39a1423e865b929223e66246"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "2b1cda3a47cea28411b87d9de33c5641"
server_key_iv: "0aec2b17883c0e3843c330627cb56e68"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.831913 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9b818b39a1423e865b929223e66246"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "2b1cda3a47cea28411b87d9de33c5641"
server_key_iv: "0aec2b17883c0e3843c330627cb56e68"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.837159 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:46.840628 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.841919 14457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:46.842089 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "1b9b818b39a1423e865b929223e66246"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "2b1cda3a47cea28411b87d9de33c5641"
server_key_iv: "0aec2b17883c0e3843c330627cb56e68"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.842284 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:46.877492 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:46.881645 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:46.881937 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:46.882905 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:46.884297 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:46.884398 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.884553 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:46.884624 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.919659 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45073
I20260620 18:32:46.919744 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45073 every 8 connection(s)
I20260620 18:32:46.920987 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:46.926330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14457
I20260620 18:32:46.926513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:46.926949 388 external_mini_cluster.cc:1428] Setting key 0136f0106de488ae3b9257b7c9167c6b
Jun 20 18:32:46 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:46.941766 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.923637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39773 (local address 127.0.97.62:34605)
0620 18:32:46.924071 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:46.924076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:46.925083 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 18:32:46.927016 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:46.927022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:46.927126 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:46.927244 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:46.929168 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.929959 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.931532 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.931960 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.932099 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.936024 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:46.936029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:46.936032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:46.936064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:46.938699 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.939420 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.939425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.939429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.939621 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.940052 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.940057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.940060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.940322 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:46.940476 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:46.941165 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:32:46.941387 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":218,"threads_started":1}
I20260620 18:32:46.942770 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.924190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:39773)
0620 18:32:46.924902 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:46.924950 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:46.926750 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:46.927488 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:46.927496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:46.928192 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:46.928999 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.929018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.930106 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.930118 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.931353 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.931364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.932187 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.932195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.932460 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.933153 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:46.933179 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:46.935773 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:46.938851 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.938860 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.938876 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.939291 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.939732 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.939737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.939741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.939925 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.940460 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:46.940468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:46.940814 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 18:32:46.941077 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 18:32:46.941629 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":303,"threads_started":1}
I20260620 18:32:46.944353 14585 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:46.944715 14585 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:46.945503 14585 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:46.946969 14262 ts_manager.cc:194] Registered new tserver with Master: 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
I20260620 18:32:46.947898 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39773
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:32:46.979962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:47.303767 14591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:47.304050 14591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:47.304257 14591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:47.312004 14591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:47.312115 14591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:47.312340 14591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:32:47.322108 14591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:47.323436 14591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:47.324848 14591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:47.338428 14599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.338483 14597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.338515 14596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:47.339443 14591 server_base.cc:1034] running on GCE node
I20260620 18:32:47.340060 14591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:47.340968 14591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:47.342219 14591 hybrid_clock.cc:648] HybridClock initialized: now 1781980367342169 us; error 53 us; skew 500 ppm
I20260620 18:32:47.342576 14591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:47 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:47.346071 14591 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:32:47.347607 14591 webserver.cc:460] Webserver started at http://127.0.97.3:42597/ using document root <none> and password file <none>
I20260620 18:32:47.348281 14591 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:47.348430 14591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:47.348722 14591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:47.351624 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-mcnm"
server_key: "60718b3c2a83ec4b317dfba27f56e2b2"
server_key_iv: "d348cc73541ddbdc638dbd69c58d9f38"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.352375 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-mcnm"
server_key: "60718b3c2a83ec4b317dfba27f56e2b2"
server_key_iv: "d348cc73541ddbdc638dbd69c58d9f38"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.357156 14591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:32:47.360453 14606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.361686 14591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:47.361858 14591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-mcnm"
server_key: "60718b3c2a83ec4b317dfba27f56e2b2"
server_key_iv: "d348cc73541ddbdc638dbd69c58d9f38"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.362054 14591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:47.380057 14591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:47.384541 14591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:47.384852 14591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:47.385870 14591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:47.387231 14591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:47.387332 14591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.387491 14591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:47.387598 14591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:32:47.422279 14591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33639
I20260620 18:32:47.422392 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33639 every 8 connection(s)
I20260620 18:32:47.423005 14451 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:47.423776 14591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:47.431864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14591
I20260620 18:32:47.432106 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:47.432543 388 external_mini_cluster.cc:1428] Setting key 4a5ba11600a9c6611b57d188557cc898
Jun 20 18:32:47 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:47.445431 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.426642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56093 (local address 127.0.97.62:34605)
0620 18:32:47.426859 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.426865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.428143 (+ 1278us) server_negotiation.cc:407] Connection header received
0620 18:32:47.430089 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.430095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.430194 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.430306 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.432014 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.432865 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.434956 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.435653 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.435793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.439761 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.439770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.439775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.439812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.442473 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.443214 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.443218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.443221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.443397 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.443910 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.443914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.443916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.444192 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.444309 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.444986 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.445164 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 18:32:47.446538 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.427072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:56093)
0620 18:32:47.427954 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.427993 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.429827 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.430551 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.430559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.431060 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.431786 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.431812 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.433018 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.433025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.434813 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.434824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.435905 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.435914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.436170 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.436891 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.436922 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.439244 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.442620 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.442627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.442641 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.443080 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.443595 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.443599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.443601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.443791 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.444314 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.444320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.444646 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.444875 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.445252 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":432,"threads_started":1}
I20260620 18:32:47.448037 14719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:47.448390 14719 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:47.449081 14719 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:47.450448 14262 ts_manager.cc:194] Registered new tserver with Master: cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639)
I20260620 18:32:47.451236 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56093
I20260620 18:32:47.460457 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:47.477260 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.465086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.0.97.62:34605)
0620 18:32:47.465271 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.465276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.465670 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:32:47.465795 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.465799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.465866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.465985 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.467246 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.468064 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.469416 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.469796 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.469897 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.472340 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.472344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.472347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.472375 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.474822 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.475469 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.475472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.475474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.475628 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.476101 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.476104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.476106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.476395 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.476588 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.476855 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.477027 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:32:47.481168 14262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:47.487895 14262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:47.521395 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.500601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.97.1:43715)
0620 18:32:47.512831 (+ 12230us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.512844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.512882 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:32:47.513021 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.513034 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.513396 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.513634 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.514573 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516335 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.518772 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.520197 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.520331 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.520577 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.520885 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.521053 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12012,"thread_start_us":11838,"threads_started":1}
I20260620 18:32:47.521445 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.500047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53366)
0620 18:32:47.510059 (+ 10012us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.510112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.510338 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.513673 (+ 3335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.513682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.513724 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.514312 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.514333 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516510 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.516515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.518431 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.518443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.520400 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.520408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.520625 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.520680 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.520840 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.521030 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9673,"thread_start_us":14251,"threads_started":1}
I20260620 18:32:47.522634 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.503427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33639 (local address 127.0.0.1:55876)
0620 18:32:47.514035 (+ 10608us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.514049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.514200 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.515072 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.515079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.515102 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.515477 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.515487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516982 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.516992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.519763 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.519778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.521789 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.521795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.522015 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.522049 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.522128 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.522259 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10510,"mutex_wait_us":150,"thread_start_us":6134,"threads_started":1}
I20260620 18:32:47.522722 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.501001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.0.97.2:45073)
0620 18:32:47.513214 (+ 12213us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.513220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.513246 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:47.513369 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.513378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.513611 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.513834 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.515232 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516776 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.519971 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.521375 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.521513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.521729 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.522213 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.522376 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12055,"thread_start_us":11890,"threads_started":1}
I20260620 18:32:47.522745 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.500343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45073 (local address 127.0.0.1:57480)
0620 18:32:47.511769 (+ 11426us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.511793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.511934 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.514103 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.514114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.514141 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.514977 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.514993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516916 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.516925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.519762 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.519778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.521872 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.521878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.522013 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.522038 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.522154 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.522304 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11293,"mutex_wait_us":131,"thread_start_us":11710,"threads_started":1}
I20260620 18:32:47.523394 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.504181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.0.97.3:33639)
0620 18:32:47.511680 (+ 7499us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.511689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.514154 (+ 2465us) server_negotiation.cc:407] Connection header received
0620 18:32:47.514368 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.514376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.514594 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.514790 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.515670 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.516835 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.519935 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.521288 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.521414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.521659 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.522899 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.523073 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6719,"thread_start_us":6519,"threads_started":1}
I20260620 18:32:47.525712 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (DEFAULT_TABLE table=test-table [id=2753fbee48f64e38950398e8833410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:47.526849 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:47.527029 14654 tablet_service.cc:1467] Processing CreateTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (DEFAULT_TABLE table=test-table [id=2753fbee48f64e38950398e8833410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:47.527029 14520 tablet_service.cc:1467] Processing CreateTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (DEFAULT_TABLE table=test-table [id=2753fbee48f64e38950398e8833410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:47.528447 14654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:47.528461 14520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:47.539181 14737 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Bootstrap starting.
I20260620 18:32:47.542265 14737 tablet_bootstrap.cc:654] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:47.543459 14738 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Bootstrap starting.
I20260620 18:32:47.543581 14737 log.cc:824] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:47.544379 14739 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Bootstrap starting.
I20260620 18:32:47.549021 14739 tablet_bootstrap.cc:654] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:47.550227 14738 tablet_bootstrap.cc:654] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:47.550617 14739 log.cc:824] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:47.551213 14738 log.cc:824] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:47.553915 14738 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: No bootstrap required, opened a new log
I20260620 18:32:47.554034 14737 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: No bootstrap required, opened a new log
I20260620 18:32:47.554034 14739 tablet_bootstrap.cc:492] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: No bootstrap required, opened a new log
I20260620 18:32:47.554244 14738 ts_tablet_manager.cc:1397] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260620 18:32:47.554301 14739 ts_tablet_manager.cc:1397] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:32:47.554301 14737 ts_tablet_manager.cc:1397] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260620 18:32:47.561672 14738 raft_consensus.cc:348] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.562173 14738 raft_consensus.cc:374] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:47.562301 14738 raft_consensus.cc:729] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9b818b39a1423e865b929223e66246, State: Initialized, Role: FOLLOWER
I20260620 18:32:47.562944 14738 consensus_queue.cc:260] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.564195 14585 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:47.564848 14738 ts_tablet_manager.cc:1428] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260620 18:32:47.565142 14739 raft_consensus.cc:348] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.565114 14737 raft_consensus.cc:348] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.565547 14737 raft_consensus.cc:374] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:47.565547 14739 raft_consensus.cc:374] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:47.565649 14739 raft_consensus.cc:729] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeb4a5f4ddd42ab9b0bc968a00b15fa, State: Initialized, Role: FOLLOWER
I20260620 18:32:47.565649 14737 raft_consensus.cc:729] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d9634705a8442ea9dcd70a0e4245e1, State: Initialized, Role: FOLLOWER
I20260620 18:32:47.566311 14739 consensus_queue.cc:260] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.566298 14737 consensus_queue.cc:260] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.567880 14719 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:47.568121 14737 ts_tablet_manager.cc:1428] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 18:32:47.568409 14739 ts_tablet_manager.cc:1428] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260620 18:32:47.610663 14744 raft_consensus.cc:482] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:47.611018 14744 raft_consensus.cc:504] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.613041 14744 leader_election.cc:288] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073), cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639)
I20260620 18:32:47.620062 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.613353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45073 (local address 127.0.97.1:48423)
0620 18:32:47.614354 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.614376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.614522 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.614873 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.614878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.614903 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.615322 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.615332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.616411 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.616417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.618003 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.618014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.619262 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.619269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.619445 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.619480 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.619592 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.619712 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":24,"thread_start_us":600,"threads_started":1}
I20260620 18:32:47.620057 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.613392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48423 (local address 127.0.97.2:45073)
0620 18:32:47.613650 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.613657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.614480 (+ 823us) server_negotiation.cc:407] Connection header received
0620 18:32:47.614660 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.614668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.614750 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.614838 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.615469 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.616275 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.618150 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.619037 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.619144 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.619219 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.619627 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.619767 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:32:47.620486 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.613854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45413 (local address 127.0.97.3:33639)
0620 18:32:47.614074 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.614081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.614716 (+ 635us) server_negotiation.cc:407] Connection header received
0620 18:32:47.614882 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.614887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.614968 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.615069 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.615678 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.616444 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.618490 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.619334 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.619466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.619592 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.620101 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.620260 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:32:47.620608 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.613704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33639 (local address 127.0.97.1:45413)
0620 18:32:47.614600 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.614617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.614739 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.615095 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.615099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.615118 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.615485 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.615496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.616639 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.616643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.618282 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.618306 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.619705 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.619712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.619886 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.619926 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.620052 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.620183 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":52,"thread_start_us":529,"threads_started":1}
I20260620 18:32:47.621480 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a18b8cc4fc042dc91cf0c8daa9bdbd0" candidate_uuid: "29d9634705a8442ea9dcd70a0e4245e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9b818b39a1423e865b929223e66246" is_pre_election: true
I20260620 18:32:47.621569 14674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a18b8cc4fc042dc91cf0c8daa9bdbd0" candidate_uuid: "29d9634705a8442ea9dcd70a0e4245e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" is_pre_election: true
I20260620 18:32:47.621989 14540 raft_consensus.cc:2454] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d9634705a8442ea9dcd70a0e4245e1 in term 0.
I20260620 18:32:47.622041 14674 raft_consensus.cc:2454] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d9634705a8442ea9dcd70a0e4245e1 in term 0.
I20260620 18:32:47.622668 14341 leader_election.cc:302] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9b818b39a1423e865b929223e66246, 29d9634705a8442ea9dcd70a0e4245e1; no voters:
I20260620 18:32:47.623052 14744 raft_consensus.cc:2781] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:47.623198 14744 raft_consensus.cc:482] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:47.623302 14744 raft_consensus.cc:3037] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:47.625487 14744 raft_consensus.cc:504] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.626096 14744 leader_election.cc:288] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073), cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639)
I20260620 18:32:47.626468 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a18b8cc4fc042dc91cf0c8daa9bdbd0" candidate_uuid: "29d9634705a8442ea9dcd70a0e4245e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9b818b39a1423e865b929223e66246"
I20260620 18:32:47.626618 14674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a18b8cc4fc042dc91cf0c8daa9bdbd0" candidate_uuid: "29d9634705a8442ea9dcd70a0e4245e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
I20260620 18:32:47.626677 14540 raft_consensus.cc:3037] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:47.626816 14674 raft_consensus.cc:3037] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:47.630008 14540 raft_consensus.cc:2454] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d9634705a8442ea9dcd70a0e4245e1 in term 1.
I20260620 18:32:47.630007 14674 raft_consensus.cc:2454] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d9634705a8442ea9dcd70a0e4245e1 in term 1.
I20260620 18:32:47.630419 14342 leader_election.cc:302] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1, cdeb4a5f4ddd42ab9b0bc968a00b15fa; no voters:
I20260620 18:32:47.630722 14744 raft_consensus.cc:2781] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:47.631135 14744 raft_consensus.cc:686] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 LEADER]: Becoming Leader. State: Replica: 29d9634705a8442ea9dcd70a0e4245e1, State: Running, Role: LEADER
I20260620 18:32:47.631649 14744 consensus_queue.cc:237] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } }
I20260620 18:32:47.635900 14262 catalog_manager.cc:5495] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "29d9634705a8442ea9dcd70a0e4245e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } health_report { overall_health: UNKNOWN } } }
W20260620 18:32:47.642268 14452 tablet.cc:2369] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:47.672120 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.666888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.97.1:43715)
0620 18:32:47.667078 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.667082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.667100 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:47.667258 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.667262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.667370 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.667490 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:47.667877 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.668596 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.670043 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.670468 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.670564 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.671006 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:47.671543 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:47.671645 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.671759 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.671822 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:32:47.673130 14586 tablet.cc:2369] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:47.675797 14720 tablet.cc:2369] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:47.680522 14674 raft_consensus.cc:1261] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Refusing update from remote peer 29d9634705a8442ea9dcd70a0e4245e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:47.680572 14540 raft_consensus.cc:1261] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Refusing update from remote peer 29d9634705a8442ea9dcd70a0e4245e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:47.681545 14748 consensus_queue.cc:1035] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:47.681915 14744 consensus_queue.cc:1035] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:47.694088 14751 mvcc.cc:205] Tried to move back new op lower bound from 7298991586009755648 to 7298991585822781440. Current Snapshot: MvccSnapshot[applied={T|T < 7298991586009755648}]
I20260620 18:32:47.707223 14262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:47.707515 14262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:47.710929 14262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Sending DeleteTablet for 3 replicas of tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0
I20260620 18:32:47.712105 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53366
I20260620 18:32:47.712168 14520 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57480
I20260620 18:32:47.712594 14654 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55876
I20260620 18:32:47.713676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14323
I20260620 18:32:47.723009 14763 tablet_replica.cc:330] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: stopping tablet replica
I20260620 18:32:47.724290 14763 raft_consensus.cc:2229] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:47.725674 14763 raft_consensus.cc:2258] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:47.727371 14762 tablet_replica.cc:330] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: stopping tablet replica
I20260620 18:32:47.727924 14762 raft_consensus.cc:2229] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:47.728513 14762 raft_consensus.cc:2258] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:47.729245 14763 ts_tablet_manager.cc:1905] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:47.730605 14762 ts_tablet_manager.cc:1905] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:47.735083 14761 tablet_replica.cc:330] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: stopping tablet replica
I20260620 18:32:47.735590 14761 raft_consensus.cc:2229] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:47.736279 14761 raft_consensus.cc:2258] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:47.737936 14762 ts_tablet_manager.cc:1918] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:47.738147 14762 log.cc:1196] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/1a18b8cc4fc042dc91cf0c8daa9bdbd0
I20260620 18:32:47.738270 14763 ts_tablet_manager.cc:1918] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:47.738610 14763 log.cc:1196] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/1a18b8cc4fc042dc91cf0c8daa9bdbd0
I20260620 18:32:47.738726 14762 ts_tablet_manager.cc:1939] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 1b9b818b39a1423e865b929223e66246: Deleting consensus metadata
I20260620 18:32:47.739638 14763 ts_tablet_manager.cc:1939] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting consensus metadata
I20260620 18:32:47.740597 14248 catalog_manager.cc:4841] TS 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (table test-table [id=2753fbee48f64e38950398e8833410fb]) successfully deleted
I20260620 18:32:47.741492 14249 catalog_manager.cc:4841] TS cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (table test-table [id=2753fbee48f64e38950398e8833410fb]) successfully deleted
Jun 20 18:32:47 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:32:47.742813 14761 ts_tablet_manager.cc:1905] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:47.749158 14761 ts_tablet_manager.cc:1918] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:47.749339 14761 log.cc:1196] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/1a18b8cc4fc042dc91cf0c8daa9bdbd0
I20260620 18:32:47.749785 14761 ts_tablet_manager.cc:1939] T 1a18b8cc4fc042dc91cf0c8daa9bdbd0 P 29d9634705a8442ea9dcd70a0e4245e1: Deleting consensus metadata
I20260620 18:32:47.751186 14246 catalog_manager.cc:4841] TS 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715): tablet 1a18b8cc4fc042dc91cf0c8daa9bdbd0 (table test-table [id=2753fbee48f64e38950398e8833410fb]) successfully deleted
I20260620 18:32:47.751753 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.716384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.97.1:43715)
0620 18:32:47.716647 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.716656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.728272 (+ 11616us) server_negotiation.cc:407] Connection header received
0620 18:32:47.728808 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.728813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.728911 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.728989 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.731311 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.732697 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.736520 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.737227 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.737319 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.743352 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.743372 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.743392 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.743449 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.748099 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.749002 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.749011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.749015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.749202 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.749690 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.749695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.749699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.750148 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.750462 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.751064 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.751314 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:32:47.752767 14446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:47.853942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14323
I20260620 18:32:47.872581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14457
Jun 20 18:32:47 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:32:47.887965 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.873392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.97.2:45073)
0620 18:32:47.873593 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.873598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.873618 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:47.873736 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.873741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.873821 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.873912 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.875120 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.875945 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.877233 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.877628 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.877765 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.882194 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.882207 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.882223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.882259 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.885076 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.885793 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.885800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.885802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.885924 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.886295 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.886302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.886305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.886715 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.887012 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.887304 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.887670 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:32:47.888661 14580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:47.977854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14457
I20260620 18:32:47.994983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14591
Jun 20 18:32:47 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:32:48.010898 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.995870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.97.3:33639)
0620 18:32:47.996050 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.996055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.996074 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:32:47.996232 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.996235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.996301 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.996393 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.997646 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.998388 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.999817 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.000293 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.000437 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.004952 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.004966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.004982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.005021 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.008237 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.008975 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.008983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.008985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.009131 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.009549 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.009553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.009555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.009899 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.010126 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.010469 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.010636 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:32:48.011561 14714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:48.100153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14591
I20260620 18:32:48.117024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14230
I20260620 18:32:48.118270 14292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:48.231156 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14230
I20260620 18:32:48.252076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--webserver_interface=127.0.97.62
--webserver_port=33311
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:48.580750 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:48.581089 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:48.581248 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:48.588346 14769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:48.588487 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:48.588603 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:48.588703 14769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:48.588789 14769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:48.598529 14769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:48.599812 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:48.601096 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:48.624162 14775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:48.624116 14776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:48.624080 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:48.625301 14769 server_base.cc:1034] running on GCE node
I20260620 18:32:48.626165 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:48.627795 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:48.629015 14769 hybrid_clock.cc:648] HybridClock initialized: now 1781980368628962 us; error 46 us; skew 500 ppm
I20260620 18:32:48.629406 14769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:48 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:48.632896 14769 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:48.634479 14769 webserver.cc:460] Webserver started at http://127.0.97.62:33311/ using document root <none> and password file <none>
I20260620 18:32:48.635157 14769 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:48.635268 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:48.641041 14769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 18:32:48.643908 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:48.645246 14769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:32:48.645426 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "57b3357baad2443ba15f692da054b775"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-mcnm"
server_key: "08f002f5a033b3cb0c42163e462bf25e"
server_key_iv: "e8e22c2f3d922f3936042c6865cdf055"
server_key_version: "encryptionkey@0"
I20260620 18:32:48.646448 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:48.680512 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:48.684860 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:48.685160 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:48.704880 14769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34605
I20260620 18:32:48.704921 14836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34605 every 8 connection(s)
I20260620 18:32:48.706507 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:48.714120 14837 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:32:48.714389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14769
I20260620 18:32:48.715500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43715
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=34749
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:48.719607 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap starting.
I20260620 18:32:48.741684 14837 log.cc:824] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:48.755589 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:48.756209 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap complete.
I20260620 18:32:48.770699 14837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:48.772034 14837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b3357baad2443ba15f692da054b775, State: Initialized, Role: FOLLOWER
I20260620 18:32:48.772804 14837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:48.773082 14837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:48.773238 14837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:48.773459 14837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:32:48.777907 14837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:48.778473 14837 leader_election.cc:302] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b3357baad2443ba15f692da054b775; no voters:
I20260620 18:32:48.779182 14837 leader_election.cc:288] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:32:48.779340 14841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:32:48.781826 14837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:48.782303 14841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 LEADER]: Becoming Leader. State: Replica: 57b3357baad2443ba15f692da054b775, State: Running, Role: LEADER
I20260620 18:32:48.782966 14841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:48.785887 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b3357baad2443ba15f692da054b775. Latest consensus state: current_term: 2 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:48.786358 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:48.787017 14842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:48.787333 14842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:48.787330 14850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:48.806517 14850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:48.808542 14850 catalog_manager.cc:1259] Loaded cluster ID: 65f0dae2796c4eda9b56ab4de0f5dec9
I20260620 18:32:48.808734 14850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:48.812230 14850 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:32:48.832860 14850 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:32:48.861105 14850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:48.863561 14850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Loaded TSK: 0
I20260620 18:32:48.865604 14850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:32:49.112138 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:49.112389 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:49.112540 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:49.119879 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:49.119978 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:49.120174 14839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:32:49.129650 14839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34749
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:49.130868 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:49.132359 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:49.146869 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.146895 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.147576 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:49.148720 14839 server_base.cc:1034] running on GCE node
I20260620 18:32:49.149415 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:49.150465 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:49.151691 14839 hybrid_clock.cc:648] HybridClock initialized: now 1781980369151639 us; error 80 us; skew 500 ppm
I20260620 18:32:49.152020 14839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:49 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:49.158874 14839 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:32:49.160457 14839 webserver.cc:460] Webserver started at http://127.0.97.1:34749/ using document root <none> and password file <none>
I20260620 18:32:49.161159 14839 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:49.161324 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:49.166957 14839 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:32:49.169915 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.171388 14839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:32:49.171597 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "29d9634705a8442ea9dcd70a0e4245e1"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "76f0ed337946d42644af6bbedb619396"
server_key_iv: "8897464de3a7e0099766614c8710e77d"
server_key_version: "encryptionkey@0"
I20260620 18:32:49.172556 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:49.196058 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:49.200330 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:49.200611 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:49.201663 14839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:49.203027 14839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:49.203148 14839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.203282 14839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:49.203377 14839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.245445 14839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43715
I20260620 18:32:49.245491 14987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43715 every 8 connection(s)
I20260620 18:32:49.246889 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:49.256000 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14839
I20260620 18:32:49.256789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45073
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=46167
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:49 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:49.275843 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.250081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:35633)
0620 18:32:49.250978 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.251038 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.253380 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.254783 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.254794 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.255561 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.256295 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.256321 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.258943 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.258949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.260469 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.260482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.264248 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.264255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.264467 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.265323 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.265357 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:49.268083 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:49.271571 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.271580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.271595 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.272016 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.272454 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.272461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.272464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.272651 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.273360 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:49.273368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:49.273701 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 18:32:49.273957 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:32:49.274501 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":347,"threads_started":1}
I20260620 18:32:49.275843 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.249875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35633 (local address 127.0.97.62:34605)
0620 18:32:49.250520 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.250542 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.251169 (+ 627us) server_negotiation.cc:407] Connection header received
0620 18:32:49.253915 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.253928 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.254272 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.254553 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.256567 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.258741 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.260631 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.261729 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.263671 (+ 1942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.268379 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.268387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.268401 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.268437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.271395 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.272169 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.272175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.272180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.272339 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.272785 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.272789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.272791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.273221 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.273432 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.274185 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.274567 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
I20260620 18:32:49.279224 14988 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:49.279616 14988 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:49.280313 14988 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:49.282616 14802 ts_manager.cc:194] Registered new tserver with Master: 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715)
I20260620 18:32:49.284739 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35633
W20260620 18:32:49.588805 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:49.589141 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:49.589250 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:49.596275 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:49.596377 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:49.596485 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:32:49.605892 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46167
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:49.607053 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:49.608354 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:49.621402 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.621448 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.621757 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:49.625017 14992 server_base.cc:1034] running on GCE node
I20260620 18:32:49.625663 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:49.626600 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:49.627868 14992 hybrid_clock.cc:648] HybridClock initialized: now 1781980369627823 us; error 45 us; skew 500 ppm
I20260620 18:32:49.628198 14992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:49 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:49.631233 14992 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:32:49.632789 14992 webserver.cc:460] Webserver started at http://127.0.97.2:46167/ using document root <none> and password file <none>
I20260620 18:32:49.633479 14992 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:49.633634 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:49.638754 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:32:49.641373 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.642611 14992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:49.642802 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "1b9b818b39a1423e865b929223e66246"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "2b1cda3a47cea28411b87d9de33c5641"
server_key_iv: "0aec2b17883c0e3843c330627cb56e68"
server_key_version: "encryptionkey@0"
I20260620 18:32:49.643819 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:49.679304 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:49.683750 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:49.684016 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:49.685000 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:49.686311 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:49.686432 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.686563 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:49.686650 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.721866 14992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45073
I20260620 18:32:49.721908 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45073 every 8 connection(s)
I20260620 18:32:49.723316 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:49.731503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 14992
I20260620 18:32:49.732241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33639
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=42597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:32:49 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:49.746712 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.726062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37171 (local address 127.0.97.62:34605)
0620 18:32:49.726248 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.726253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.727769 (+ 1516us) server_negotiation.cc:407] Connection header received
0620 18:32:49.729514 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.729519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.729602 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.729698 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.731132 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.732016 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.734538 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.734994 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.735104 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.739500 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.739505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.739508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.739589 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.743211 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.744031 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.744040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.744046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.744207 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.744644 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.744652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.744657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.745114 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.745266 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.746009 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.746268 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:49.747469 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.726694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:37171)
0620 18:32:49.727571 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.727618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.729263 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.729930 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.729938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.730429 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.730982 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.730999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.732181 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.732187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.734382 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.734400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.735226 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.735235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.735571 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.736333 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.736373 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:49.739262 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:49.743358 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.743364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.743378 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.743908 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.744323 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.744327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.744329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.744517 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.745278 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:49.745285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:49.745599 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 18:32:49.745863 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:32:49.746246 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":316,"threads_started":1}
I20260620 18:32:49.749111 15120 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:49.749431 15120 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:49.750227 15120 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:49.751768 14802 ts_manager.cc:194] Registered new tserver with Master: 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
I20260620 18:32:49.752584 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37171
W20260620 18:32:50.063779 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:50.064040 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:50.064208 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:50.071444 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:50.071614 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:50.071805 15123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:32:50.081614 15123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42597
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:50.082950 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:50.084332 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:50.097149 15128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:50.097195 15131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:50.097224 15129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:50.098434 15123 server_base.cc:1034] running on GCE node
I20260620 18:32:50.099040 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:50.100138 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:50.101370 15123 hybrid_clock.cc:648] HybridClock initialized: now 1781980370101326 us; error 40 us; skew 500 ppm
I20260620 18:32:50.101703 15123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:50 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:50.104923 15123 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:32:50.106420 15123 webserver.cc:460] Webserver started at http://127.0.97.3:42597/ using document root <none> and password file <none>
I20260620 18:32:50.107101 15123 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:50.107262 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:50.112680 15123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:50.115305 15138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:50.116837 15123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:32:50.117023 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-mcnm"
server_key: "60718b3c2a83ec4b317dfba27f56e2b2"
server_key_iv: "d348cc73541ddbdc638dbd69c58d9f38"
server_key_version: "encryptionkey@0"
I20260620 18:32:50.118006 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:50.130174 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:50.133989 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:50.134263 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:50.135174 15123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:50.136507 15123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:50.136627 15123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:50.136765 15123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:50.136857 15123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:50.170890 15123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33639
I20260620 18:32:50.170979 15250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33639 every 8 connection(s)
I20260620 18:32:50.172379 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:50.179548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15123
Jun 20 18:32:50 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:32:50.196595 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.175022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47309 (local address 127.0.97.62:34605)
0620 18:32:50.175215 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.175220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.176598 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 18:32:50.178475 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.178480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.178553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.178667 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.181138 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.182246 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.184575 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.185251 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.185395 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.189256 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.189263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.189268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.189313 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.193184 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.193918 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.193923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.193926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.194056 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.194641 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.194650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.194652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.194947 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.195137 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.195959 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.196261 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":72}
I20260620 18:32:50.196611 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.180720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.0.97.62:34605)
0620 18:32:50.181546 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.181551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.181566 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:32:50.181609 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.181612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.181684 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.181771 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.183587 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.184664 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.186360 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.186807 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.186939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.190185 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.190201 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.190207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.190277 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.193727 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.194483 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.194491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.194494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.194612 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.194973 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.194979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.194983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.195256 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.195500 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.195838 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.196084 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":607,"threads_started":1}
I20260620 18:32:50.198024 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.175610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:47309)
0620 18:32:50.176409 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.176453 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.178189 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.179031 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.179046 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.180055 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:50.180865 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.180915 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.182492 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.182499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.184420 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.184436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.185491 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.185496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.185733 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.186480 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:50.186504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:50.188999 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:50.193335 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.193342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.193356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.193820 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.194280 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.194284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.194286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.194481 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.195131 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:50.195139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:50.195493 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.195844 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.196579 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":277,"threads_started":1}
I20260620 18:32:50.199404 15251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34605
I20260620 18:32:50.199740 15251 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:50.200480 15251 heartbeater.cc:511] Master 127.0.97.62:34605 requested a full tablet report, sending...
I20260620 18:32:50.201884 14801 ts_manager.cc:194] Registered new tserver with Master: cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639)
I20260620 18:32:50.202757 14801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47309
I20260620 18:32:50.211695 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:50.218914 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.212587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.0.97.62:34605)
0620 18:32:50.212821 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.212830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.213250 (+ 420us) server_negotiation.cc:407] Connection header received
0620 18:32:50.213421 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.213425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.213566 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.213705 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:50.214135 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.214846 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.216426 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.216948 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.217048 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.217582 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:50.218205 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:50.218318 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.218505 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.218654 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:32:50.220021 14801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:50.227373 14801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:50.258419 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.238883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33639 (local address 127.0.0.1:55890)
0620 18:32:50.242241 (+ 3358us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.242291 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.242519 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.244022 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.244033 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.244090 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.244688 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.244707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.251056 (+ 6349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.251064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.254249 (+ 3185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.254272 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.257235 (+ 2963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.257241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.257462 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.257518 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.257681 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.257922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3024,"thread_start_us":16789,"threads_started":1}
I20260620 18:32:50.258922 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.239149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.0.97.3:33639)
0620 18:32:50.243276 (+ 4127us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.243288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.243330 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:32:50.243443 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.243454 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.243778 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.243992 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.245113 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.249387 (+ 4274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.255074 (+ 5687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.257024 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.257184 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.257421 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.258089 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.258314 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3929,"thread_start_us":3749,"threads_started":1}
I20260620 18:32:50.262952 15186 tablet_service.cc:1467] Processing CreateTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 (DEFAULT_TABLE table=test-table [id=49db5b313e74422d9fe5e3e378d5c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.264377 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce8835b32b402da4e8b3d4a8c65c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.264784 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.239364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45073 (local address 127.0.0.1:57498)
0620 18:32:50.255953 (+ 16589us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.255983 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.256132 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.257482 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.257487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.257516 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.257982 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.257998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.260394 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.260399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.262045 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.262067 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.263904 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.263912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.264091 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.264124 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.264221 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.264353 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16437,"mutex_wait_us":39,"thread_start_us":10175,"threads_started":1}
I20260620 18:32:50.265204 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.239377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.0.97.1:43715)
0620 18:32:50.257097 (+ 17720us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.257108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.257135 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:32:50.257227 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.257234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.257431 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.257814 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.258396 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.260044 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.262604 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.264120 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.264265 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.264483 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.264737 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.264893 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17501,"thread_start_us":17358,"threads_started":1}
I20260620 18:32:50.265630 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.239854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.97.2:45073)
0620 18:32:50.256607 (+ 16753us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.256617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.256657 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:32:50.256788 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.256795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.257065 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.257270 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.258320 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.260203 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.262199 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.263684 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.263818 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.264005 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.265176 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.265331 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16615,"thread_start_us":16512,"threads_started":1}
I20260620 18:32:50.265180 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.238916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53382)
0620 18:32:50.254650 (+ 15734us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.254676 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.254812 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.257646 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.257652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.257698 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.258191 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.258205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.260212 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.260217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.262432 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.262447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.264332 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.264338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.264524 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.264566 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.264681 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.264809 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15536,"mutex_wait_us":26,"thread_start_us":3328,"threads_started":1}
I20260620 18:32:50.269420 14923 tablet_service.cc:1467] Processing CreateTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 (DEFAULT_TABLE table=test-table [id=49db5b313e74422d9fe5e3e378d5c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.269392 15055 tablet_service.cc:1467] Processing CreateTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 (DEFAULT_TABLE table=test-table [id=49db5b313e74422d9fe5e3e378d5c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.270601 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce8835b32b402da4e8b3d4a8c65c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.270699 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce8835b32b402da4e8b3d4a8c65c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.280299 15267 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Bootstrap starting.
I20260620 18:32:50.285017 15269 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Bootstrap starting.
I20260620 18:32:50.285702 15267 tablet_bootstrap.cc:654] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.287353 15267 log.cc:824] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.287743 14988 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:50.287971 15269 tablet_bootstrap.cc:654] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.289091 15269 log.cc:824] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.290328 15268 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Bootstrap starting.
I20260620 18:32:50.296000 15268 tablet_bootstrap.cc:654] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.297042 15267 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: No bootstrap required, opened a new log
I20260620 18:32:50.297091 15269 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: No bootstrap required, opened a new log
I20260620 18:32:50.297256 15268 log.cc:824] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.297359 15269 ts_tablet_manager.cc:1397] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 18:32:50.297389 15267 ts_tablet_manager.cc:1397] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260620 18:32:50.300292 15268 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: No bootstrap required, opened a new log
I20260620 18:32:50.300590 15268 ts_tablet_manager.cc:1397] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 18:32:50.304698 15267 raft_consensus.cc:348] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.305065 15267 raft_consensus.cc:374] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.305181 15267 raft_consensus.cc:729] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeb4a5f4ddd42ab9b0bc968a00b15fa, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.305827 15267 consensus_queue.cc:260] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.307834 15269 raft_consensus.cc:348] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.308313 15269 raft_consensus.cc:374] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.308420 15269 raft_consensus.cc:729] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9b818b39a1423e865b929223e66246, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.308532 15251 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:50.309142 15267 ts_tablet_manager.cc:1428] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260620 18:32:50.309031 15269 consensus_queue.cc:260] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.310137 15120 heartbeater.cc:503] Master 127.0.97.62:34605 was elected leader, sending a full tablet report...
I20260620 18:32:50.311975 15269 ts_tablet_manager.cc:1428] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:32:50.313321 15268 raft_consensus.cc:348] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.313738 15268 raft_consensus.cc:374] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.313891 15268 raft_consensus.cc:729] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d9634705a8442ea9dcd70a0e4245e1, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.314440 15268 consensus_queue.cc:260] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.315989 15268 ts_tablet_manager.cc:1428] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 18:32:50.350515 15273 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:50.350857 15273 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.352639 15273 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
I20260620 18:32:50.358088 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.353059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33663 (local address 127.0.97.1:43715)
0620 18:32:50.353274 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.353282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.353304 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:50.353382 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.353390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.353472 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.353571 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.354005 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.354721 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.356460 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.357221 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.357372 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.357473 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.357728 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.357853 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:50.358115 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.352861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.97.3:33663)
0620 18:32:50.353098 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.353120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.353252 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.353599 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.353605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.353643 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.353847 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.353855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.354862 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.354871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.356279 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.356292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.357380 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.357386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.357565 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.357587 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.357692 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.357821 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:32:50.358899 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.353323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42433 (local address 127.0.97.2:45073)
0620 18:32:50.353503 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.353511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.353986 (+ 475us) server_negotiation.cc:407] Connection header received
0620 18:32:50.354180 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.354184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.354239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.354338 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.354813 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.355491 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.357176 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.357938 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.358097 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.358190 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.358537 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.358662 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:50.359006 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.353020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45073 (local address 127.0.97.3:42433)
0620 18:32:50.353883 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.353897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.354031 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.354347 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.354351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.354374 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.354673 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.354681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.355697 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.355703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.357031 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.357042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.358152 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.358158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.358304 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.358356 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.358509 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.358638 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":1,"thread_start_us":507,"threads_started":1}
I20260620 18:32:50.359344 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce8835b32b402da4e8b3d4a8c65c71" candidate_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d9634705a8442ea9dcd70a0e4245e1" is_pre_election: true
I20260620 18:32:50.359946 14943 raft_consensus.cc:2454] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdeb4a5f4ddd42ab9b0bc968a00b15fa in term 0.
I20260620 18:32:50.360045 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce8835b32b402da4e8b3d4a8c65c71" candidate_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9b818b39a1423e865b929223e66246" is_pre_election: true
I20260620 18:32:50.360605 15075 raft_consensus.cc:2454] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdeb4a5f4ddd42ab9b0bc968a00b15fa in term 0.
I20260620 18:32:50.360682 15139 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1, cdeb4a5f4ddd42ab9b0bc968a00b15fa; no voters:
I20260620 18:32:50.361148 15273 raft_consensus.cc:2781] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:50.361313 15273 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:50.361442 15273 raft_consensus.cc:3037] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.363835 15273 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.364439 15273 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [CANDIDATE]: Term 1 election: Requested vote from peers 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
I20260620 18:32:50.364892 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce8835b32b402da4e8b3d4a8c65c71" candidate_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9b818b39a1423e865b929223e66246"
I20260620 18:32:50.364921 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce8835b32b402da4e8b3d4a8c65c71" candidate_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d9634705a8442ea9dcd70a0e4245e1"
I20260620 18:32:50.365095 15075 raft_consensus.cc:3037] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.365109 14943 raft_consensus.cc:3037] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.367602 15075 raft_consensus.cc:2454] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdeb4a5f4ddd42ab9b0bc968a00b15fa in term 1.
I20260620 18:32:50.367645 14943 raft_consensus.cc:2454] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdeb4a5f4ddd42ab9b0bc968a00b15fa in term 1.
I20260620 18:32:50.368033 15141 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa; no voters:
I20260620 18:32:50.368359 15273 raft_consensus.cc:2781] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:50.368790 15273 raft_consensus.cc:686] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 LEADER]: Becoming Leader. State: Replica: cdeb4a5f4ddd42ab9b0bc968a00b15fa, State: Running, Role: LEADER
I20260620 18:32:50.369297 15273 consensus_queue.cc:237] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:50.373872 14802 catalog_manager.cc:5495] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa reported cstate change: term changed from 0 to 1, leader changed from <none> to cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:50.410224 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.404887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.0.97.3:33639)
0620 18:32:50.405078 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.405085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.405106 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:32:50.405306 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.405309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.405398 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.405524 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:50.405904 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.406653 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.408168 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.408576 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.408698 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.409149 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:50.409654 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:50.409769 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.409913 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.409982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:50.418766 14943 raft_consensus.cc:1261] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Refusing update from remote peer cdeb4a5f4ddd42ab9b0bc968a00b15fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:50.418766 15075 raft_consensus.cc:1261] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Refusing update from remote peer cdeb4a5f4ddd42ab9b0bc968a00b15fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:50.419896 15277 consensus_queue.cc:1035] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:50.420301 15273 consensus_queue.cc:1035] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:32:50.423975 15252 tablet.cc:2369] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:50.432226 15280 mvcc.cc:205] Tried to move back new op lower bound from 7298991597223784448 to 7298991597037092864. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597223784448}]
I20260620 18:32:50.433403 15281 mvcc.cc:205] Tried to move back new op lower bound from 7298991597223784448 to 7298991597037092864. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597223784448}]
I20260620 18:32:50.435300 15282 mvcc.cc:205] Tried to move back new op lower bound from 7298991597223784448 to 7298991597037092864. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597223784448}]
I20260620 18:32:50.447034 14802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:50.447309 14802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:50.450994 14802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Sending DeleteTablet for 3 replicas of tablet cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:50.452191 14923 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53382
I20260620 18:32:50.452283 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260620 18:32:50.452626 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14839
I20260620 18:32:50.453166 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce8835b32b402da4e8b3d4a8c65c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55890
I20260620 18:32:50.457038 15291 tablet_replica.cc:330] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: stopping tablet replica
I20260620 18:32:50.457548 15291 raft_consensus.cc:2229] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:50.461030 15291 raft_consensus.cc:2258] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.463649 15291 ts_tablet_manager.cc:1905] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.465360 15290 tablet_replica.cc:330] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: stopping tablet replica
I20260620 18:32:50.465792 15290 raft_consensus.cc:2229] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:50.466300 15290 raft_consensus.cc:2258] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.469600 15290 ts_tablet_manager.cc:1905] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.471405 15291 ts_tablet_manager.cc:1918] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:50.471650 15291 log.cc:1196] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:50.471838 15289 tablet_replica.cc:330] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: stopping tablet replica
I20260620 18:32:50.472289 15289 raft_consensus.cc:2229] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:50.472345 15291 ts_tablet_manager.cc:1939] T cfce8835b32b402da4e8b3d4a8c65c71 P cdeb4a5f4ddd42ab9b0bc968a00b15fa: Deleting consensus metadata
I20260620 18:32:50.474334 14789 catalog_manager.cc:4841] TS cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): tablet cfce8835b32b402da4e8b3d4a8c65c71 (table test-table [id=49db5b313e74422d9fe5e3e378d5c255]) successfully deleted
I20260620 18:32:50.475481 15289 raft_consensus.cc:2258] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.477780 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.453801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.0.97.1:43715)
0620 18:32:50.454028 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.454037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.454071 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:32:50.454148 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.454156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.454296 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.454424 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.456496 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.458077 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.460723 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.461649 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.461824 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.465284 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.465306 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.465330 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.465391 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.471677 (+ 6286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.472574 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.472586 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.472591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.472741 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.473142 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.473147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.473150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.476264 (+ 3114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.476591 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.476923 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.477223 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:32:50.477934 15289 ts_tablet_manager.cc:1905] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.478696 14983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:50.482755 15290 ts_tablet_manager.cc:1918] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:50.482964 15290 log.cc:1196] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:50.483552 15290 ts_tablet_manager.cc:1939] T cfce8835b32b402da4e8b3d4a8c65c71 P 1b9b818b39a1423e865b929223e66246: Deleting consensus metadata
I20260620 18:32:50.485335 14788 catalog_manager.cc:4841] TS 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): tablet cfce8835b32b402da4e8b3d4a8c65c71 (table test-table [id=49db5b313e74422d9fe5e3e378d5c255]) successfully deleted
I20260620 18:32:50.569262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14839
W20260620 18:32:50.593259 14786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43715 (error 108)
I20260620 18:32:50.593575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14992
I20260620 18:32:50.595577 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.595054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53408)
0620 18:32:50.595248 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.595367 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 18:32:50.595849 14786 catalog_manager.cc:4584] TS 29d9634705a8442ea9dcd70a0e4245e1 (127.0.97.1:43715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfce8835b32b402da4e8b3d4a8c65c71: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
I20260620 18:32:50.610059 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.594968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.97.2:45073)
0620 18:32:50.595179 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.595186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.595204 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:50.595269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.595274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.595356 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.595463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.597456 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.598174 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.600164 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.600564 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.600693 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.604206 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.604219 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.604234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.604269 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.607126 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.608036 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.608043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.608046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.608185 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.608609 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.608612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.608614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.608967 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.609194 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.609584 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.609749 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:50.610828 15115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:50.649551 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.648926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53418)
0620 18:32:50.649203 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.649337 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 18:32:50.650796 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.650375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53432)
0620 18:32:50.650552 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.650657 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:32:50.701908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14992
I20260620 18:32:50.711403 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.710871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53436)
0620 18:32:50.711130 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.711217 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 18:32:50.712647 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.712215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53440)
0620 18:32:50.712434 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.712523 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 18:32:50.725478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15123
I20260620 18:32:50.739346 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.726426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.0.97.3:33639)
0620 18:32:50.726634 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.726639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.726664 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:50.726737 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.726741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.726798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.726907 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.728309 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.729147 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.730494 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.730924 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.731064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.733600 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.733611 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.733623 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.733667 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.736591 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.737300 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.737306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.737308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.737464 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.737825 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.737834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.737837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.738174 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.738503 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.738775 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.738991 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:32:50.740255 15246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:50.793503 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.792965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53444)
0620 18:32:50.793211 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.793318 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 18:32:50.794709 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.794300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53446)
0620 18:32:50.794502 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.794591 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:32:50.866182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15123
I20260620 18:32:50.886030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 14769
I20260620 18:32:50.887316 14832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:51.000959 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.000272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43715 (local address 127.0.0.1:53452)
0620 18:32:51.000606 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.000721 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 18:32:51.000968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 14769
I20260620 18:32:51.022701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--webserver_interface=127.0.97.62
--webserver_port=33311
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:51.344601 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:51.344961 15297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:51.345135 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:51.352572 15297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:51.352730 15297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:51.352852 15297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:51.352952 15297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:51.353042 15297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:51.363201 15297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:51.364645 15297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:51.366087 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:51.378579 15303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:51.378589 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:51.378750 15302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:51.379988 15297 server_base.cc:1034] running on GCE node
I20260620 18:32:51.380955 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:51.382542 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:51.383791 15297 hybrid_clock.cc:648] HybridClock initialized: now 1781980371383737 us; error 63 us; skew 500 ppm
I20260620 18:32:51.384186 15297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:51 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:51.387715 15297 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:32:51.389350 15297 webserver.cc:460] Webserver started at http://127.0.97.62:33311/ using document root <none> and password file <none>
I20260620 18:32:51.390102 15297 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:51.390275 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:51.397178 15297 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:32:51.400215 15312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:51.401707 15297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:51.401908 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "57b3357baad2443ba15f692da054b775"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-mcnm"
server_key: "08f002f5a033b3cb0c42163e462bf25e"
server_key_iv: "e8e22c2f3d922f3936042c6865cdf055"
server_key_version: "encryptionkey@0"
I20260620 18:32:51.403012 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:51.417701 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:51.422078 15297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:51.422379 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:51.440665 15297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34605
I20260620 18:32:51.440699 15363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34605 every 8 connection(s)
I20260620 18:32:51.442206 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:51.444195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15297
I20260620 18:32:51.445007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43715
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=34749
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:51.449205 15364 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:32:51.453390 15364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap starting.
I20260620 18:32:51.484097 15364 log.cc:824] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:51.504729 15364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:51.505488 15364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775: Bootstrap complete.
I20260620 18:32:51.520403 15364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:51.521651 15364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b3357baad2443ba15f692da054b775, State: Initialized, Role: FOLLOWER
I20260620 18:32:51.522343 15364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:51.522610 15364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:51.522727 15364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:51.522897 15364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:32:51.528297 15364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:51.528857 15364 leader_election.cc:302] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b3357baad2443ba15f692da054b775; no voters:
I20260620 18:32:51.529520 15364 leader_election.cc:288] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:32:51.529672 15368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:32:51.531004 15368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [term 3 LEADER]: Becoming Leader. State: Replica: 57b3357baad2443ba15f692da054b775, State: Running, Role: LEADER
I20260620 18:32:51.531529 15364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:51.531639 15368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } }
I20260620 18:32:51.534283 15370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b3357baad2443ba15f692da054b775. Latest consensus state: current_term: 3 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:51.534650 15370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:51.535223 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57b3357baad2443ba15f692da054b775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b3357baad2443ba15f692da054b775" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34605 } } }
I20260620 18:32:51.535547 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b3357baad2443ba15f692da054b775 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:51.544593 15375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:51.555204 15375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:51.556720 15375 catalog_manager.cc:1259] Loaded cluster ID: 65f0dae2796c4eda9b56ab4de0f5dec9
I20260620 18:32:51.556875 15375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:32:51.558804 15375 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5425e369ad google::LogMessage::Fail() at ??:0
@ 0x7f5425e3988a google::LogMessage::SendToLog() at ??:0
@ 0x7f5425e364de google::LogMessage::Flush() at ??:0
@ 0x7f5425e37a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f543b869512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f543b866976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f543b8ee610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f543b8ee439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5428a82345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5428a8591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5428a85799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5428a60eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f542bd3a6db start_thread at ??:0
@ 0x7f5424cd061f clone at ??:0
W20260620 18:32:51.875982 15366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:51.876272 15366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:51.876374 15366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:51.883747 15366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:51.883862 15366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:51.883984 15366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:32:51.893927 15366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34749
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:51.895166 15366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:51.896528 15366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:51.910578 15405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:51.910542 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:51.910542 15406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:51.911624 15366 server_base.cc:1034] running on GCE node
I20260620 18:32:51.912202 15366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:51.913164 15366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:51.914371 15366 hybrid_clock.cc:648] HybridClock initialized: now 1781980371914324 us; error 36 us; skew 500 ppm
I20260620 18:32:51.914713 15366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:51 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:51.917943 15366 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:32:51.919407 15366 webserver.cc:460] Webserver started at http://127.0.97.1:34749/ using document root <none> and password file <none>
I20260620 18:32:51.920135 15366 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:51.920264 15366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:51.925590 15366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:32:51.928457 15415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:51.929827 15366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:51.930035 15366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "29d9634705a8442ea9dcd70a0e4245e1"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "76f0ed337946d42644af6bbedb619396"
server_key_iv: "8897464de3a7e0099766614c8710e77d"
server_key_version: "encryptionkey@0"
I20260620 18:32:51.931034 15366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:51.954070 15366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:51.958257 15366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:51.958565 15366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:51.959578 15366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:51.961367 15422 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 18:32:51.967259 15366 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:32:51.967406 15366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260620 18:32:51.967603 15366 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:32:51.970643 15366 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:32:51.970752 15366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 18:32:51.971083 15422 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Bootstrap starting.
I20260620 18:32:51.979254 15422 log.cc:824] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:51.996680 15422 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:51.997522 15422 tablet_bootstrap.cc:492] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Bootstrap complete.
I20260620 18:32:51.998099 15422 ts_tablet_manager.cc:1397] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260620 18:32:52.008373 15422 raft_consensus.cc:348] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:52.009980 15422 raft_consensus.cc:729] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d9634705a8442ea9dcd70a0e4245e1, State: Initialized, Role: FOLLOWER
I20260620 18:32:52.010643 15422 consensus_queue.cc:260] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:52.012266 15422 ts_tablet_manager.cc:1428] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:32:52.016264 15366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43715
I20260620 18:32:52.016335 15530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43715 every 8 connection(s)
I20260620 18:32:52.017452 15366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:52.021893 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.020132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:41293)
0620 18:32:52.020869 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.021063 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":304,"threads_started":1}
W20260620 18:32:52.022596 15531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34605 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34605: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
I20260620 18:32:52.023358 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.023043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:51575)
0620 18:32:52.023192 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.023259 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:32:52.024206 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.023880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:32777)
0620 18:32:52.024021 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.024101 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:32:52.024451 15531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:32:52.025880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15366
I20260620 18:32:52.026487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45073
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=46167
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:52.345520 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:52.345790 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:52.345955 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:52.353139 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:52.353250 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:52.353451 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:32:52.363111 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46167
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:52.364466 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:52.365747 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:52.378715 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:52.378808 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:52.378747 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:52.379822 15534 server_base.cc:1034] running on GCE node
I20260620 18:32:52.380417 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:52.381346 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:52.382582 15534 hybrid_clock.cc:648] HybridClock initialized: now 1781980372382536 us; error 52 us; skew 500 ppm
I20260620 18:32:52.382908 15534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:52 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:52.386138 15534 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:32:52.387727 15534 webserver.cc:460] Webserver started at http://127.0.97.2:46167/ using document root <none> and password file <none>
I20260620 18:32:52.388418 15534 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:52.388564 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:52.393953 15534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:32:52.397186 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.398555 15534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:52.398743 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "1b9b818b39a1423e865b929223e66246"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-mcnm"
server_key: "2b1cda3a47cea28411b87d9de33c5641"
server_key_iv: "0aec2b17883c0e3843c330627cb56e68"
server_key_version: "encryptionkey@0"
I20260620 18:32:52.399816 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:52.411721 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:52.416307 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:52.416658 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:52.417681 15534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:52.419049 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:52.419169 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.419299 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:52.419394 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.454114 15534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45073
I20260620 18:32:52.454198 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45073 every 8 connection(s)
I20260620 18:32:52.455587 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:52.457729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15534
I20260620 18:32:52.458305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33639
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34605
--webserver_port=42597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:52.460860 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.458569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:39431)
0620 18:32:52.459433 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.459699 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":289,"threads_started":1}
W20260620 18:32:52.461575 15662 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34605 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34605: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
I20260620 18:32:52.462538 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.462137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:42811)
0620 18:32:52.462310 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.462395 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:32:52.463599 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.463187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:45497)
0620 18:32:52.463347 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.463435 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:32:52.463896 15662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:32:52.799661 15665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:52.799938 15665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:52.800029 15665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:52.807595 15665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:52.807708 15665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:52.807818 15665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:32:52.817322 15665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42597
--tserver_master_addrs=127.0.97.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:32:52.818533 15665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:52.819774 15665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:52.833132 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:52.833122 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:52.833171 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:52.834509 15665 server_base.cc:1034] running on GCE node
I20260620 18:32:52.835233 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:52.836346 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:52.837577 15665 hybrid_clock.cc:648] HybridClock initialized: now 1781980372837531 us; error 35 us; skew 500 ppm
I20260620 18:32:52.837963 15665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:52 dist-test-slave-mcnm krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:52.841300 15665 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:32:52.842770 15665 webserver.cc:460] Webserver started at http://127.0.97.3:42597/ using document root <none> and password file <none>
I20260620 18:32:52.843470 15665 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:52.843636 15665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:52.849099 15665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 18:32:52.852535 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.854337 15665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:32:52.854507 15665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-mcnm"
server_key: "60718b3c2a83ec4b317dfba27f56e2b2"
server_key_iv: "d348cc73541ddbdc638dbd69c58d9f38"
server_key_version: "encryptionkey@0"
I20260620 18:32:52.855502 15665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:52.882591 15665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:52.886693 15665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:52.886991 15665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:52.888049 15665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:52.889367 15665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:52.889488 15665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.889619 15665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:52.889707 15665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:52.926717 15665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33639
I20260620 18:32:52.926826 15792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33639 every 8 connection(s)
I20260620 18:32:52.928356 15665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:52.933275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15665
I20260620 18:32:52.933867 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.931486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:36747)
0620 18:32:52.932311 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.932557 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":302,"threads_started":1}
W20260620 18:32:52.934594 15793 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34605 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34605: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
I20260620 18:32:52.935708 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.935320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:38457)
0620 18:32:52.935458 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.935590 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:32:52.936662 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.936325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:54577)
0620 18:32:52.936480 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.936555 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:32:52.936906 15793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:32:53.025842 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.024969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:35033)
0620 18:32:53.025522 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:53.025637 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":240,"threads_started":1}
I20260620 18:32:53.465173 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.464400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:47391)
0620 18:32:53.464846 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:53.464948 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260620 18:32:53.475617 15799 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:53.475914 15799 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:53.477521 15799 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
Jun 20 18:32:53 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 18:32:53 dist-test-slave-mcnm krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:32:53.498317 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.477721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33639 (local address 127.0.97.1:39745)
0620 18:32:53.477972 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:53.478016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:53.478619 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:53.482204 (+ 3585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:53.482211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:53.482907 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:53.483690 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:53.483710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.485191 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.485194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:53.486594 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:53.486605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.487579 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.487587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:53.487842 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:53.488593 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:53.488618 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:53.491273 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:53.494864 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:53.494872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:53.494875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:53.495381 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:53.495848 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:53.495852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:53.495856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:53.496030 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:53.496769 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:53.496776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:53.497069 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 18:32:53.497348 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 18:32:53.497865 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":194,"spinlock_wait_cycles":14592}
I20260620 18:32:53.498318 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.479212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39745 (local address 127.0.97.3:33639)
0620 18:32:53.479754 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:53.479765 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:53.479793 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:53.481477 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:53.481485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:53.481884 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:53.482212 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:53.483888 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.484999 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.486728 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.487336 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.487560 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:53.491567 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:53.491583 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:53.491595 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:53.491635 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:53.494721 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:53.495532 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:53.495540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:53.495550 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:53.495727 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:53.496146 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:53.496153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:53.496156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:53.496626 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:53.496811 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:53.497573 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 18:32:53.497911 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260620 18:32:53.498412 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.478046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51375 (local address 127.0.97.2:45073)
0620 18:32:53.479115 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:53.479130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:53.479172 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:32:53.481060 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:53.481072 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:53.481549 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:53.481812 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:53.483884 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.484999 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.486723 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.487341 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.487566 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:53.491409 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:53.491419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:53.491436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:53.491477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:53.494626 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:53.495568 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:53.495578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:53.495591 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:53.495742 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:53.496199 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:53.496212 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:53.496215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:53.496704 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:53.496894 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:53.497582 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 18:32:53.497957 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":737,"threads_started":1}
I20260620 18:32:53.498471 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.478023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45073 (local address 127.0.97.1:51375)
0620 18:32:53.478763 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:53.478786 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:53.478908 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:53.481990 (+ 3082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:53.482001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:53.482917 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:53.483690 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:53.483709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.485190 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.485194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:53.486591 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:53.486602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:53.487587 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:53.487594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:53.487843 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:53.488654 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:53.488671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:53.491093 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:53.494828 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:53.494838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:53.494852 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:53.495396 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:53.495876 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:53.495881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:53.495886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:53.496073 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:53.496849 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:53.496855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:53.497069 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 18:32:53.497350 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:32:53.498022 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":352,"spinlock_wait_cycles":25088,"thread_start_us":334,"threads_started":1}
W20260620 18:32:53.501932 15419 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
W20260620 18:32:53.502203 15418 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:53.502368 15418 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1; no voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa
I20260620 18:32:53.502879 15799 raft_consensus.cc:2726] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:32:53.938352 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:53.937473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:57513)
0620 18:32:53.937980 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:53.938104 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":220,"threads_started":1}
I20260620 18:32:54.027562 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:54.026781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:40793)
0620 18:32:54.027195 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:54.027322 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260620 18:32:54.466876 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:54.466088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:60175)
0620 18:32:54.466607 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:54.466705 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":282,"threads_started":1}
I20260620 18:32:54.940058 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:54.939236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:58693)
0620 18:32:54.939810 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:54.939890 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":236,"threads_started":1}
I20260620 18:32:55.029487 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.028596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:54433)
0620 18:32:55.029146 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.029264 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":287,"threads_started":1}
I20260620 18:32:55.130429 15808 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:55.130640 15808 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:55.131278 15808 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
W20260620 18:32:55.132336 15419 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
W20260620 18:32:55.132488 15418 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:55.132623 15418 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1; no voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa
I20260620 18:32:55.132934 15808 raft_consensus.cc:2726] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:32:55.468743 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.468018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:51873)
0620 18:32:55.468481 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.468564 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260620 18:32:55.941758 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.940985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:45483)
0620 18:32:55.941517 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.941603 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":292,"threads_started":1}
I20260620 18:32:56.031212 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.030390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:52505)
0620 18:32:56.030944 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:56.031024 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":338,"threads_started":1}
I20260620 18:32:56.470438 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.469696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:41619)
0620 18:32:56.470166 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:56.470244 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260620 18:32:56.943411 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.942684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:40313)
0620 18:32:56.943141 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:56.943229 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260620 18:32:57.033103 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.032271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:42371)
0620 18:32:57.032820 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.032907 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260620 18:32:57.312424 15815 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:57.312692 15815 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:32:57.313382 15815 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
W20260620 18:32:57.314380 15418 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
W20260620 18:32:57.314602 15419 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:32:57.314772 15419 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1; no voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa
I20260620 18:32:57.315052 15815 raft_consensus.cc:2726] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:32:57.472121 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.471379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:37687)
0620 18:32:57.471871 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.471960 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
I20260620 18:32:57.945159 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.944338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:40063)
0620 18:32:57.944888 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.944973 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":263,"threads_started":1}
I20260620 18:32:58.034739 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.033995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:39205)
0620 18:32:58.034502 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.034577 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260620 18:32:58.473819 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.473013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:47021)
0620 18:32:58.473543 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.473629 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":264,"threads_started":1}
I20260620 18:32:58.946784 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.946015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:51899)
0620 18:32:58.946535 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.946615 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260620 18:32:59.036533 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:59.035754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:44455)
0620 18:32:59.036201 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:59.036329 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260620 18:32:59.475667 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:59.474799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:54857)
0620 18:32:59.475303 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:59.475415 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":233,"threads_started":1}
I20260620 18:32:59.948529 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:59.947762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:50111)
0620 18:32:59.948267 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:59.948347 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":235,"threads_started":1}
I20260620 18:33:00.038164 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:00.037400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:33489)
0620 18:33:00.037883 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:00.037983 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260620 18:33:00.477458 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:00.476731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:39623)
0620 18:33:00.477192 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:00.477284 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
I20260620 18:33:00.950297 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:00.949544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:39967)
0620 18:33:00.950001 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:00.950084 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260620 18:33:01.039952 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:01.039134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:41513)
0620 18:33:01.039663 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:01.039775 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260620 18:33:01.362448 15828 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:33:01.362648 15828 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:33:01.363552 15828 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
W20260620 18:33:01.364552 15418 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
W20260620 18:33:01.364806 15419 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:33:01.364936 15419 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1; no voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa
I20260620 18:33:01.365229 15828 raft_consensus.cc:2726] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:33:01.479349 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:01.478518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:35449)
0620 18:33:01.479071 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:01.479177 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260620 18:33:01.952250 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:01.951376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:54157)
0620 18:33:01.951985 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:01.952079 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":327,"threads_started":1}
I20260620 18:33:02.041880 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:02.041124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:52339)
0620 18:33:02.041622 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:02.041711 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260620 18:33:02.481141 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:02.480386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:39147)
0620 18:33:02.480875 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:02.480963 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260620 18:33:02.953831 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:02.953171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:34093)
0620 18:33:02.953586 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:02.953660 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260620 18:33:03.043689 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:03.042853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:47297)
0620 18:33:03.043306 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:03.043388 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
I20260620 18:33:03.482923 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:03.482158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:34459)
0620 18:33:03.482660 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:03.482743 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
I20260620 18:33:03.955744 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:03.954817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:45541)
0620 18:33:03.955452 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:03.955588 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":388,"threads_started":1}
I20260620 18:33:04.045419 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.044714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:49273)
0620 18:33:04.045178 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:04.045254 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260620 18:33:04.484684 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.483862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:45963)
0620 18:33:04.484387 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:04.484482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":261,"threads_started":1}
I20260620 18:33:04.957587 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.956801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:41233)
0620 18:33:04.957317 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:04.957406 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260620 18:33:05.047237 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:05.046460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:43651)
0620 18:33:05.046970 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:05.047055 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
I20260620 18:33:05.486631 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:05.485761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:44493)
0620 18:33:05.486312 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:05.486435 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":245,"threads_started":1}
I20260620 18:33:05.959669 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:05.958652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:47491)
0620 18:33:05.959309 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:05.959395 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":369,"threads_started":1}
I20260620 18:33:06.048997 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:06.048284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:33535)
0620 18:33:06.048713 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:06.048820 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260620 18:33:06.488417 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:06.487655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:51367)
0620 18:33:06.488121 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:06.488206 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260620 18:33:06.961472 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:06.960667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:35351)
0620 18:33:06.961177 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:06.961265 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":241,"threads_started":1}
I20260620 18:33:07.050908 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.050149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.1:59445)
0620 18:33:07.050612 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.050729 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260620 18:33:07.490311 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.489550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.2:36445)
0620 18:33:07.490042 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.490136 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260620 18:33:07.679178 15848 raft_consensus.cc:482] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:33:07.679432 15848 raft_consensus.cc:504] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdeb4a5f4ddd42ab9b0bc968a00b15fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33639 } } peers { permanent_uuid: "29d9634705a8442ea9dcd70a0e4245e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43715 } } peers { permanent_uuid: "1b9b818b39a1423e865b929223e66246" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45073 } }
I20260620 18:33:07.680141 15848 leader_election.cc:288] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639), 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073)
W20260620 18:33:07.681156 15419 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cdeb4a5f4ddd42ab9b0bc968a00b15fa (127.0.97.3:33639): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
W20260620 18:33:07.681342 15418 leader_election.cc:341] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9b818b39a1423e865b929223e66246 (127.0.97.2:45073): Not found: Tablet not found: cfce8835b32b402da4e8b3d4a8c65c71
I20260620 18:33:07.681473 15418 leader_election.cc:302] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d9634705a8442ea9dcd70a0e4245e1; no voters: 1b9b818b39a1423e865b929223e66246, cdeb4a5f4ddd42ab9b0bc968a00b15fa
I20260620 18:33:07.681766 15848 raft_consensus.cc:2726] T cfce8835b32b402da4e8b3d4a8c65c71 P 29d9634705a8442ea9dcd70a0e4245e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:33:07.941190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15366
I20260620 18:33:07.961931 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.942420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.0.97.1:43715)
0620 18:33:07.943206 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:07.943219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:07.943255 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:33:07.943344 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:07.943355 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:07.943692 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:07.943877 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:07.945420 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.946669 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.948040 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.948572 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.948674 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.956249 (+ 7575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:07.956261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:07.956275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:07.956315 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:07.959275 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:07.960020 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:07.960027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:07.960030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:07.960170 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:07.960521 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:07.960524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:07.960526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:07.960867 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:07.961086 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:07.961436 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 18:33:07.961640 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":386,"threads_started":1}
I20260620 18:33:07.963392 15525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:07.963428 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.962676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34605 (local address 127.0.97.3:39845)
0620 18:33:07.963191 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.963285 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":230,"threads_started":1}
I20260620 18:33:08.052487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15366
I20260620 18:33:08.070279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15534
I20260620 18:33:08.084797 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.071278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.0.97.2:45073)
0620 18:33:08.071965 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.071972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.071999 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:33:08.072059 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.072063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.072127 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.072251 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.073812 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.074662 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.076133 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.076586 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.076741 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.079369 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:08.079374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:08.079378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:08.079412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:08.082058 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:08.082753 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:08.082761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:08.082769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:08.082905 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:08.083277 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:08.083283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:08.083287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:08.083596 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:08.083854 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:08.084174 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 18:33:08.084390 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":325,"threads_started":1}
I20260620 18:33:08.085475 15657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:08.168296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15534
I20260620 18:33:08.185875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15665
I20260620 18:33:08.199321 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.186718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.0.97.3:33639)
0620 18:33:08.187285 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.187289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.187303 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:33:08.187355 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.187359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.187424 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.187574 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.188991 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.189744 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.190993 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.191476 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.191670 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.194096 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:08.194101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:08.194104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:08.194136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:08.196715 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:08.197378 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:08.197385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:08.197389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:08.197526 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:08.197905 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:08.197911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:08.197914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:08.198184 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:08.198463 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:08.198738 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:33:08.198944 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260620 18:33:08.200078 15788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:08.282032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15665
2026-06-20T18:33:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24304 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:33:08 dist-test-slave-mcnm krb5kdc[15859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:33:08 dist-test-slave-mcnm krb5kdc[15859](info): set up 2 sockets
Jun 20 18:33:08 dist-test-slave-mcnm krb5kdc[15859](info): commencing operation
krb5kdc: starting...
W20260620 18:33:09.396189 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:33:09 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:33:09.634836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:09.958894 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:09.959232 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:09.959403 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:09.966624 15872 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:33:09.966751 15872 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:33:09.966846 15872 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:33:09.966931 15872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:09.967011 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:09.967114 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:09.967197 15872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:09.967276 15872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:09.976985 15872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:09.978296 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:09.979681 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:09.991842 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:09.991932 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:09.991955 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:09.992743 15872 server_base.cc:1034] running on GCE node
I20260620 18:33:09.993494 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:09.994874 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:09.996124 15872 hybrid_clock.cc:648] HybridClock initialized: now 1781980389996063 us; error 28 us; skew 500 ppm
I20260620 18:33:09.996502 15872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:09 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:09.999857 15872 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:33:10.001336 15872 webserver.cc:460] Webserver started at http://127.0.97.62:46071/ using document root <none> and password file <none>
I20260620 18:33:10.002048 15872 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:10.002161 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:10.002509 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:10.005524 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "9383b3527f5e4ac0a853303dcd1a3a7b"
format_stamp: "Formatted at 2026-06-20 18:33:09 on dist-test-slave-mcnm"
server_key: "b59f2679bc4d0f0ed86383775329329e"
server_key_iv: "a2fd45fd329d34616ca361565e855849"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.006325 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "9383b3527f5e4ac0a853303dcd1a3a7b"
format_stamp: "Formatted at 2026-06-20 18:33:09 on dist-test-slave-mcnm"
server_key: "b59f2679bc4d0f0ed86383775329329e"
server_key_iv: "a2fd45fd329d34616ca361565e855849"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.012367 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 18:33:10.016068 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:10.017710 15872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:33:10.017907 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "9383b3527f5e4ac0a853303dcd1a3a7b"
format_stamp: "Formatted at 2026-06-20 18:33:09 on dist-test-slave-mcnm"
server_key: "b59f2679bc4d0f0ed86383775329329e"
server_key_iv: "a2fd45fd329d34616ca361565e855849"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.018146 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:10.033797 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:10.038120 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:10.038399 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:10.057617 15872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44853
I20260620 18:33:10.057650 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44853 every 8 connection(s)
I20260620 18:33:10.058967 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:10.062640 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:10.067755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15872
I20260620 18:33:10.067983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:33:10.068718 388 external_mini_cluster.cc:1428] Setting key 9fb50c5396672524f249a95d790318b4
I20260620 18:33:10.074090 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap starting.
I20260620 18:33:10.077188 15939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:10.078155 15939 log.cc:824] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:10.081130 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: No bootstrap required, opened a new log
Jun 20 18:33:10 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:10.091806 15939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:10.092294 15939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:10.092406 15939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Initialized, Role: FOLLOWER
I20260620 18:33:10.092860 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.070865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.0.97.62:44853)
0620 18:33:10.071691 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:10.071703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:10.071745 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:33:10.073429 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:10.073451 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:10.073828 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:10.074215 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:10.075741 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.077002 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.078416 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.079307 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.079564 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.083938 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:10.083948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:10.083972 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:10.084021 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:10.088100 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:10.088801 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:10.088810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:10.088818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:10.089034 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:10.089507 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:10.089513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:10.089517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:10.090287 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:10.090605 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:10.091013 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:33:10.091449 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":327,"threads_started":1}
I20260620 18:33:10.093094 15939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:10.093393 15939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:10.093554 15939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:10.093740 15939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:10.096527 15939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:10.096953 15939 leader_election.cc:302] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9383b3527f5e4ac0a853303dcd1a3a7b; no voters:
I20260620 18:33:10.097641 15939 leader_election.cc:288] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:33:10.097774 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:10.098752 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 LEADER]: Becoming Leader. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Running, Role: LEADER
I20260620 18:33:10.099355 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:10.100046 15939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:10.102346 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:10.102723 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:10.102447 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9383b3527f5e4ac0a853303dcd1a3a7b. Latest consensus state: current_term: 1 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:10.102833 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:10.104384 15949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:10.109592 15949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:10.118279 15949 catalog_manager.cc:1347] Generated new cluster ID: d582f04d84a54c8b9960c5b3dca9d3df
I20260620 18:33:10.118449 15949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:10.141589 15949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:33:10.143576 15949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:10.157634 15949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Generated new TSK 0
I20260620 18:33:10.158723 15949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:33:10.229197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:10.557272 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:10.557547 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:10.557703 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:10.565006 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:10.565112 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:10.565330 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:33:10.575067 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:10.576491 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:10.577821 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:10.592191 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:10.592136 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:10.592118 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:10.593809 15965 server_base.cc:1034] running on GCE node
I20260620 18:33:10.594445 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:10.595430 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:10.596695 15965 hybrid_clock.cc:648] HybridClock initialized: now 1781980390596656 us; error 34 us; skew 500 ppm
I20260620 18:33:10.597031 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:10 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:10.600266 15965 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:33:10.601912 15965 webserver.cc:460] Webserver started at http://127.0.97.1:40273/ using document root <none> and password file <none>
I20260620 18:33:10.602636 15965 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:10.602763 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:10.603087 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:10.606267 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e85526cfdf849758c4b115a9bfd0cde"
format_stamp: "Formatted at 2026-06-20 18:33:10 on dist-test-slave-mcnm"
server_key: "1dae4f0f4c5135e8f5235fbcb7594242"
server_key_iv: "83a40d7210079b663b537fcce26c924e"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.607074 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e85526cfdf849758c4b115a9bfd0cde"
format_stamp: "Formatted at 2026-06-20 18:33:10 on dist-test-slave-mcnm"
server_key: "1dae4f0f4c5135e8f5235fbcb7594242"
server_key_iv: "83a40d7210079b663b537fcce26c924e"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.612179 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:33:10.615707 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:10.617125 15965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:33:10.617352 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "5e85526cfdf849758c4b115a9bfd0cde"
format_stamp: "Formatted at 2026-06-20 18:33:10 on dist-test-slave-mcnm"
server_key: "1dae4f0f4c5135e8f5235fbcb7594242"
server_key_iv: "83a40d7210079b663b537fcce26c924e"
server_key_version: "encryptionkey@0"
I20260620 18:33:10.617524 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:10.643693 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:10.648028 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:10.648335 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:10.649302 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:10.650624 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:10.650725 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:10.650877 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:10.650954 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:10.685087 15965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40205
I20260620 18:33:10.685141 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40205 every 8 connection(s)
I20260620 18:33:10.686378 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:10.694288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 15965
I20260620 18:33:10.694509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:33:10.695142 388 external_mini_cluster.cc:1428] Setting key 37846525667b1fc2df0975969d736868
Jun 20 18:33:10 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:10.708025 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.688935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50167 (local address 127.0.97.62:44853)
0620 18:33:10.689369 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:10.689374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:10.690357 (+ 983us) server_negotiation.cc:407] Connection header received
0620 18:33:10.692210 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:10.692216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:10.692307 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:10.692428 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:10.693925 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.694804 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.696739 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.697392 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.697529 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.700995 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:10.701004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:10.701012 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:10.701081 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:10.704670 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:10.705573 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:10.705578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:10.705580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:10.705707 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:10.706264 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:10.706267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:10.706269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:10.706554 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:10.706701 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:10.707589 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 18:33:10.707773 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":214,"threads_started":1}
I20260620 18:33:10.709439 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.689495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:50167)
0620 18:33:10.690198 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:10.690234 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:10.691939 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:10.692693 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:10.692701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:10.693194 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:10.693744 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.693761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.694952 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.694958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.696580 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.696594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.697603 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.697608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.697796 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.698477 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:10.698498 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:10.700770 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:10.704796 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:10.704802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:10.704815 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:10.705427 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:10.705841 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:10.705846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:10.705852 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:10.706118 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:10.706685 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:10.706696 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:10.707124 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 18:33:10.707423 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:33:10.708004 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":280,"threads_started":1}
I20260620 18:33:10.711007 16093 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:10.711325 16093 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:10.712086 16093 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
I20260620 18:33:10.714190 15904 ts_manager.cc:194] Registered new tserver with Master: 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205)
I20260620 18:33:10.715719 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50167
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:33:10.751801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:11.089634 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:11.089962 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:11.090144 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:11.097427 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:11.097574 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:11.097760 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:33:11.107632 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:11.108877 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:11.110176 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:11.123610 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:11.123571 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:11.123643 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:11.124430 16099 server_base.cc:1034] running on GCE node
I20260620 18:33:11.125036 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:11.126001 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:11.127259 16099 hybrid_clock.cc:648] HybridClock initialized: now 1781980391127207 us; error 57 us; skew 500 ppm
I20260620 18:33:11.127669 16099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:11 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:11.130867 16099 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:33:11.132376 16099 webserver.cc:460] Webserver started at http://127.0.97.2:45629/ using document root <none> and password file <none>
I20260620 18:33:11.133001 16099 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:11.133097 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:11.133350 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:11.136157 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f46527d9ad44093b016a0c5100c8b1f"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "fcf02f4daecb4f31a0f5ac766d0da97f"
server_key_iv: "cd5b0e796d3618c412076b41b7831c92"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.136844 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f46527d9ad44093b016a0c5100c8b1f"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "fcf02f4daecb4f31a0f5ac766d0da97f"
server_key_iv: "cd5b0e796d3618c412076b41b7831c92"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.141695 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:11.145213 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.146540 16099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:33:11.146667 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "2f46527d9ad44093b016a0c5100c8b1f"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "fcf02f4daecb4f31a0f5ac766d0da97f"
server_key_iv: "cd5b0e796d3618c412076b41b7831c92"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.146835 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:11.172677 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:11.176872 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:11.177158 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:11.178107 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:11.179394 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:11.179492 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.179649 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:11.179721 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.213739 16099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42637
I20260620 18:33:11.213836 16226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42637 every 8 connection(s)
I20260620 18:33:11.215013 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:11.217157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16099
I20260620 18:33:11.217363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:33:11.217928 388 external_mini_cluster.cc:1428] Setting key d6da056784e1651b8adf865c47278355
Jun 20 18:33:11 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:11.235673 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.217902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54203 (local address 127.0.97.62:44853)
0620 18:33:11.218386 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.218392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.219460 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 18:33:11.221527 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.221531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.221592 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.221708 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.223356 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.224167 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.225697 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.226169 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.226315 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.229803 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:11.229808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:11.229811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:11.229844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:11.232572 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.233306 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.233342 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.233345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.233470 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.233915 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.233922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.233930 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.234182 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:11.234358 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.235021 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.235258 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260620 18:33:11.236702 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.218466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:54203)
0620 18:33:11.219293 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.219344 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.221115 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.222021 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.222029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.222623 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:11.223176 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.223192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.224330 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.224343 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.225543 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.225559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.226402 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.226410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.226635 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.227333 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:11.227355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:11.229581 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:11.232718 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:11.232728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:11.232745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:11.233174 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:11.233583 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:11.233590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:11.233594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:11.233776 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:11.234311 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:11.234320 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:11.234660 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 18:33:11.234941 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:33:11.235464 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":369,"threads_started":1}
I20260620 18:33:11.238286 16227 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:11.238638 16227 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:11.239571 16227 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:33:11.241199 15904 ts_manager.cc:194] Registered new tserver with Master: 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:11.242029 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:33:11.271304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:11.594655 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:11.594928 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:11.595027 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:11.602412 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:11.602538 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:11.602655 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:33:11.612550 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:11.613824 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:11.615089 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:11.630530 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:11.630496 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:11.630575 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:11.631364 16233 server_base.cc:1034] running on GCE node
I20260620 18:33:11.631989 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:11.632881 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:11.634140 16233 hybrid_clock.cc:648] HybridClock initialized: now 1781980391634092 us; error 32 us; skew 500 ppm
I20260620 18:33:11.634502 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:11 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:11.637957 16233 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:33:11.639575 16233 webserver.cc:460] Webserver started at http://127.0.97.3:36559/ using document root <none> and password file <none>
I20260620 18:33:11.640450 16233 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:11.640587 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:11.640898 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:11.643867 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "5fa08d45f83248c39105a008fbc4e90a"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "33ea8baa94ed48e34468a2c65fb112d3"
server_key_iv: "09c0d3ce774d63059748cf2d438d8f56"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.644764 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5fa08d45f83248c39105a008fbc4e90a"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "33ea8baa94ed48e34468a2c65fb112d3"
server_key_iv: "09c0d3ce774d63059748cf2d438d8f56"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.650065 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:11.653682 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.654947 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:33:11.655118 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "5fa08d45f83248c39105a008fbc4e90a"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "33ea8baa94ed48e34468a2c65fb112d3"
server_key_iv: "09c0d3ce774d63059748cf2d438d8f56"
server_key_version: "encryptionkey@0"
I20260620 18:33:11.655359 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:11.669544 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:11.674096 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:11.674417 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:11.675436 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:11.676821 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:11.676927 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.677076 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:11.677160 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:11.711845 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36129
I20260620 18:33:11.711894 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36129 every 8 connection(s)
I20260620 18:33:11.713119 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:11.714803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16233
I20260620 18:33:11.714998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:33:11.715582 388 external_mini_cluster.cc:1428] Setting key 19c0a180bec762c96e4288ec759b38f9
I20260620 18:33:11.718703 16093 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
Jun 20 18:33:11 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:11.734262 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.717024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41957 (local address 127.0.97.62:44853)
0620 18:33:11.717200 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.717206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.717429 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:33:11.719661 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.719667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.719745 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.719862 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.721421 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.722327 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.723986 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.724392 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.724485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.727885 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:11.727890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:11.727893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:11.727925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:11.731226 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.732114 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.732119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.732122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.732255 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.732707 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.732711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.732713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.732990 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:11.733146 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.733831 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.733999 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:33:11.735208 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.716406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:41957)
0620 18:33:11.717261 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.717312 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.719378 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.720055 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.720063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.720675 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:11.721277 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.721296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.722505 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.722509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.723865 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.723877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.724577 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.724582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.724797 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.725435 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:11.725457 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:11.727675 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:11.731385 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:11.731400 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:11.731413 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:11.731922 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:11.732382 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:11.732386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:11.732388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:11.732569 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:11.733151 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:11.733159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:11.733477 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:33:11.733715 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:33:11.734080 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":381,"threads_started":1}
I20260620 18:33:11.736718 16361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:11.736986 16361 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:11.737694 16361 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
I20260620 18:33:11.739015 15904 ts_manager.cc:194] Registered new tserver with Master: 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129)
I20260620 18:33:11.739828 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41957
I20260620 18:33:11.743680 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:33:11.761868 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.748379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.97.62:44853)
0620 18:33:11.748561 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.748566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.749069 (+ 503us) server_negotiation.cc:407] Connection header received
0620 18:33:11.749183 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.749187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.749256 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.749344 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.750819 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.751690 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.753171 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.753582 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.753675 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.756417 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:11.756422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:11.756426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:11.756458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:11.759182 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.759981 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.759985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.759987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.760100 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:11.760519 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:11.760526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:11.760530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:11.760887 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:11.761107 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.761462 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.761622 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:33:11.765986 15904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:33:11.772787 15904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:33:11.799156 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.785594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.0.97.3:36129)
0620 18:33:11.790234 (+ 4640us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.790248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.790291 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:33:11.790420 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.790431 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.790756 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.790991 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:11.792184 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.793808 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.796521 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.797970 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.798100 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.798359 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.798639 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.798806 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4409,"thread_start_us":4212,"threads_started":1}
I20260620 18:33:11.799211 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.785405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36129 (local address 127.0.0.1:55362)
0620 18:33:11.787920 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.787969 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.788197 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.791375 (+ 3178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.791387 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.791433 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:11.791976 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.791996 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.793927 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.793932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.796118 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.796138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.798153 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.798159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.798385 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.798448 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 18:33:11.798580 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:33:11.798776 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903,"mutex_wait_us":4,"thread_start_us":5315,"threads_started":1}
I20260620 18:33:11.800441 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.785737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40205 (local address 127.0.0.1:37134)
0620 18:33:11.790439 (+ 4702us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.790460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.790598 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.792875 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.792882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.792918 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:11.793494 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.793534 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.795477 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.795482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.797745 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.797760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.799623 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.799631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.799800 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.799839 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:33:11.799910 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:33:11.800034 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4551,"mutex_wait_us":21,"thread_start_us":1419,"threads_started":1}
I20260620 18:33:11.801373 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.787255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42637 (local address 127.0.0.1:55932)
0620 18:33:11.793514 (+ 6259us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.793533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.793718 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.794416 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.794421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.794445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:11.794847 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.794859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.796407 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.796411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.798462 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.798476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.800614 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.800623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.800785 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.800814 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:33:11.800912 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:33:11.801029 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6129,"mutex_wait_us":5,"thread_start_us":5933,"threads_started":1}
I20260620 18:33:11.801452 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.787887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.0.97.2:42637)
0620 18:33:11.791330 (+ 3443us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.791341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.793625 (+ 2284us) server_negotiation.cc:407] Connection header received
0620 18:33:11.793935 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.793942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.794195 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.794393 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:11.795018 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.796203 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.798651 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.800088 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.800191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.800374 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.800974 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.801133 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3259,"thread_start_us":3115,"threads_started":1}
I20260620 18:33:11.802282 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.786354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.97.1:40205)
0620 18:33:11.792134 (+ 5780us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.792146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.792180 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:33:11.792298 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.792309 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.792612 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.793722 (+ 1110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:11.793947 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.795327 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.797925 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.799371 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.800696 (+ 1325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.800920 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:11.800986 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:33:11.801153 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5564,"thread_start_us":5362,"threads_started":1}
I20260620 18:33:11.803789 16296 tablet_service.cc:1467] Processing CreateTablet for tablet 440a649a2f204a09939dc913823ac72e (DEFAULT_TABLE table=test-table [id=9d57f33129b943859a026f009b253ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:11.805128 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440a649a2f204a09939dc913823ac72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:11.805150 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 440a649a2f204a09939dc913823ac72e (DEFAULT_TABLE table=test-table [id=9d57f33129b943859a026f009b253ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:11.806035 16028 tablet_service.cc:1467] Processing CreateTablet for tablet 440a649a2f204a09939dc913823ac72e (DEFAULT_TABLE table=test-table [id=9d57f33129b943859a026f009b253ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:11.806663 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440a649a2f204a09939dc913823ac72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:11.807416 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440a649a2f204a09939dc913823ac72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:11.816254 16379 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Bootstrap starting.
I20260620 18:33:11.819185 16379 tablet_bootstrap.cc:654] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:11.820247 16379 log.cc:824] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:11.823940 16379 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: No bootstrap required, opened a new log
I20260620 18:33:11.824234 16379 ts_tablet_manager.cc:1397] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:33:11.824436 16380 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Bootstrap starting.
I20260620 18:33:11.824584 16382 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Bootstrap starting.
I20260620 18:33:11.827241 16382 tablet_bootstrap.cc:654] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:11.827917 16380 tablet_bootstrap.cc:654] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:11.828420 16382 log.cc:824] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:11.829106 16380 log.cc:824] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:11.831089 16382 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: No bootstrap required, opened a new log
I20260620 18:33:11.831399 16382 ts_tablet_manager.cc:1397] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260620 18:33:11.833787 16380 tablet_bootstrap.cc:492] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: No bootstrap required, opened a new log
I20260620 18:33:11.834052 16380 ts_tablet_manager.cc:1397] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260620 18:33:11.835364 16379 raft_consensus.cc:348] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.835877 16379 raft_consensus.cc:374] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:11.835983 16379 raft_consensus.cc:729] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa08d45f83248c39105a008fbc4e90a, State: Initialized, Role: FOLLOWER
I20260620 18:33:11.836594 16379 consensus_queue.cc:260] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.837741 16361 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
I20260620 18:33:11.838375 16379 ts_tablet_manager.cc:1428] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 18:33:11.840479 16382 raft_consensus.cc:348] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.840844 16382 raft_consensus.cc:374] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:11.840966 16382 raft_consensus.cc:729] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e85526cfdf849758c4b115a9bfd0cde, State: Initialized, Role: FOLLOWER
I20260620 18:33:11.841559 16382 consensus_queue.cc:260] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.843355 16382 ts_tablet_manager.cc:1428] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s
I20260620 18:33:11.844040 16380 raft_consensus.cc:348] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.844521 16380 raft_consensus.cc:374] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:11.844641 16380 raft_consensus.cc:729] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f46527d9ad44093b016a0c5100c8b1f, State: Initialized, Role: FOLLOWER
I20260620 18:33:11.845211 16380 consensus_queue.cc:260] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:11.846364 16227 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
I20260620 18:33:11.847044 16380 ts_tablet_manager.cc:1428] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260620 18:33:11.938803 16094 tablet.cc:2369] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:33:11.964805 16362 tablet.cc:2369] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:33:11.966988 16228 tablet.cc:2369] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:33:12.065361 16386 raft_consensus.cc:482] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:33:12.065699 16386 raft_consensus.cc:504] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:12.067452 16386 leader_election.cc:288] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129), 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:12.074766 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.067979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49853 (local address 127.0.97.2:42637)
0620 18:33:12.068230 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.068239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.068881 (+ 642us) server_negotiation.cc:407] Connection header received
0620 18:33:12.069193 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.069199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.069277 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.069372 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:12.070002 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.071097 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.073029 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.073808 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.073975 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.074081 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.074376 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.074505 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:33:12.074787 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.068357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59603 (local address 127.0.97.3:36129)
0620 18:33:12.068558 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.068567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.069153 (+ 586us) server_negotiation.cc:407] Connection header received
0620 18:33:12.069392 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.069397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.069475 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.069574 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:12.070249 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.071296 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.073062 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.073823 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.073992 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.074109 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.074403 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.074530 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:33:12.074949 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.067829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36129 (local address 127.0.97.1:59603)
0620 18:33:12.069067 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:12.069084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:12.069252 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:12.069622 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:12.069627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:12.069654 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:12.070070 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.070082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.071433 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.071438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.072901 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.072912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.074099 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.074105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.074250 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.074272 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:33:12.074377 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:33:12.074499 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":4,"thread_start_us":938,"threads_started":1}
I20260620 18:33:12.075186 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.067740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42637 (local address 127.0.97.1:49853)
0620 18:33:12.068765 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:12.068787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:12.069015 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:12.069409 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:12.069414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:12.069435 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:12.069853 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.069865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.071239 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.071244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.072871 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.072883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.074053 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.074060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.074219 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.074243 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:33:12.074350 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:33:12.074486 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":51,"thread_start_us":683,"threads_started":1}
I20260620 18:33:12.076041 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440a649a2f204a09939dc913823ac72e" candidate_uuid: "5e85526cfdf849758c4b115a9bfd0cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa08d45f83248c39105a008fbc4e90a" is_pre_election: true
I20260620 18:33:12.076165 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440a649a2f204a09939dc913823ac72e" candidate_uuid: "5e85526cfdf849758c4b115a9bfd0cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f46527d9ad44093b016a0c5100c8b1f" is_pre_election: true
I20260620 18:33:12.076565 16316 raft_consensus.cc:2454] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e85526cfdf849758c4b115a9bfd0cde in term 0.
I20260620 18:33:12.076705 16182 raft_consensus.cc:2454] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e85526cfdf849758c4b115a9bfd0cde in term 0.
I20260620 18:33:12.077234 15981 leader_election.cc:302] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e85526cfdf849758c4b115a9bfd0cde, 5fa08d45f83248c39105a008fbc4e90a; no voters:
I20260620 18:33:12.077608 16386 raft_consensus.cc:2781] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:33:12.077765 16386 raft_consensus.cc:482] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:33:12.077881 16386 raft_consensus.cc:3037] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:12.080354 16386 raft_consensus.cc:504] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:12.081008 16386 leader_election.cc:288] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [CANDIDATE]: Term 1 election: Requested vote from peers 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129), 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:12.081478 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440a649a2f204a09939dc913823ac72e" candidate_uuid: "5e85526cfdf849758c4b115a9bfd0cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa08d45f83248c39105a008fbc4e90a"
I20260620 18:33:12.081478 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440a649a2f204a09939dc913823ac72e" candidate_uuid: "5e85526cfdf849758c4b115a9bfd0cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f46527d9ad44093b016a0c5100c8b1f"
I20260620 18:33:12.081681 16182 raft_consensus.cc:3037] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:12.081684 16316 raft_consensus.cc:3037] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:12.084851 16182 raft_consensus.cc:2454] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e85526cfdf849758c4b115a9bfd0cde in term 1.
I20260620 18:33:12.084848 16316 raft_consensus.cc:2454] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e85526cfdf849758c4b115a9bfd0cde in term 1.
I20260620 18:33:12.085225 15984 leader_election.cc:302] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f46527d9ad44093b016a0c5100c8b1f, 5e85526cfdf849758c4b115a9bfd0cde; no voters:
I20260620 18:33:12.085552 16386 raft_consensus.cc:2781] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:12.085990 16386 raft_consensus.cc:686] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 1 LEADER]: Becoming Leader. State: Replica: 5e85526cfdf849758c4b115a9bfd0cde, State: Running, Role: LEADER
I20260620 18:33:12.086444 16386 consensus_queue.cc:237] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:12.090961 15904 catalog_manager.cc:5495] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5e85526cfdf849758c4b115a9bfd0cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260620 18:33:12.162539 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.157003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.0.97.1:40205)
0620 18:33:12.157187 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.157192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.157349 (+ 157us) server_negotiation.cc:407] Connection header received
0620 18:33:12.157518 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.157522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.157599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.157744 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:33:12.158133 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.158905 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.160507 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.160912 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.161014 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.161548 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:33:12.162061 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:33:12.162176 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.162279 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.162343 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:33:12.170706 16182 raft_consensus.cc:1261] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Refusing update from remote peer 5e85526cfdf849758c4b115a9bfd0cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:12.170709 16316 raft_consensus.cc:1261] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Refusing update from remote peer 5e85526cfdf849758c4b115a9bfd0cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:12.171787 16390 consensus_queue.cc:1035] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:12.172142 16386 consensus_queue.cc:1035] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:12.183693 16395 mvcc.cc:205] Tried to move back new op lower bound from 7298991686321246208 to 7298991685989761024. Current Snapshot: MvccSnapshot[applied={T|T < 7298991686321246208}]
I20260620 18:33:12.188228 16394 mvcc.cc:205] Tried to move back new op lower bound from 7298991686321246208 to 7298991685989761024. Current Snapshot: MvccSnapshot[applied={T|T < 7298991686321246208}]
I20260620 18:33:12.195917 15903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:12.196204 15903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:12.199702 15903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Sending DeleteTablet for 3 replicas of tablet 440a649a2f204a09939dc913823ac72e
I20260620 18:33:12.200863 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet 440a649a2f204a09939dc913823ac72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55362
I20260620 18:33:12.201164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15965
I20260620 18:33:12.201418 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 440a649a2f204a09939dc913823ac72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55932
I20260620 18:33:12.201463 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet 440a649a2f204a09939dc913823ac72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37134
I20260620 18:33:12.214571 16404 tablet_replica.cc:330] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: stopping tablet replica
I20260620 18:33:12.215013 16404 raft_consensus.cc:2229] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:33:12.215758 16404 raft_consensus.cc:2258] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:12.216118 16402 tablet_replica.cc:330] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: stopping tablet replica
I20260620 18:33:12.216519 16402 raft_consensus.cc:2229] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:12.216967 16402 raft_consensus.cc:2258] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:12.219846 16402 ts_tablet_manager.cc:1905] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:12.220193 16403 tablet_replica.cc:330] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: stopping tablet replica
I20260620 18:33:12.223410 16404 ts_tablet_manager.cc:1905] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:12.226836 16402 ts_tablet_manager.cc:1918] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:12.226994 16402 log.cc:1196] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/440a649a2f204a09939dc913823ac72e
I20260620 18:33:12.227479 16402 ts_tablet_manager.cc:1939] T 440a649a2f204a09939dc913823ac72e P 5fa08d45f83248c39105a008fbc4e90a: Deleting consensus metadata
I20260620 18:33:12.229198 15888 catalog_manager.cc:4841] TS 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129): tablet 440a649a2f204a09939dc913823ac72e (table test-table [id=9d57f33129b943859a026f009b253ee8]) successfully deleted
I20260620 18:33:12.230652 16404 ts_tablet_manager.cc:1918] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:12.230825 16404 log.cc:1196] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/440a649a2f204a09939dc913823ac72e
I20260620 18:33:12.231380 16404 ts_tablet_manager.cc:1939] T 440a649a2f204a09939dc913823ac72e P 5e85526cfdf849758c4b115a9bfd0cde: Deleting consensus metadata
I20260620 18:33:12.222388 16403 raft_consensus.cc:2229] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:12.231962 16403 raft_consensus.cc:2258] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:12.234457 16403 ts_tablet_manager.cc:1905] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:33:12 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:33:12.240985 15890 catalog_manager.cc:4841] TS 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205): tablet 440a649a2f204a09939dc913823ac72e (table test-table [id=9d57f33129b943859a026f009b253ee8]) successfully deleted
I20260620 18:33:12.242151 16403 ts_tablet_manager.cc:1918] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:12.242313 16403 log.cc:1196] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/440a649a2f204a09939dc913823ac72e
I20260620 18:33:12.242841 16403 ts_tablet_manager.cc:1939] T 440a649a2f204a09939dc913823ac72e P 2f46527d9ad44093b016a0c5100c8b1f: Deleting consensus metadata
I20260620 18:33:12.244565 15891 catalog_manager.cc:4841] TS 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637): tablet 440a649a2f204a09939dc913823ac72e (table test-table [id=9d57f33129b943859a026f009b253ee8]) successfully deleted
I20260620 18:33:12.245474 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.202513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.97.1:40205)
0620 18:33:12.202740 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.202748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.211426 (+ 8678us) server_negotiation.cc:407] Connection header received
0620 18:33:12.211854 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.211861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.211958 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.212041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.216142 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.217652 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.231875 (+ 14223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.232600 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.232709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.238751 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:12.238766 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:12.238783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:12.238822 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:12.242421 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.243197 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.243206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.243211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.243373 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.243811 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.243818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.243826 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.244167 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:12.244484 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.244880 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.245121 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:33:12.246304 16088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:12.345193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15965
I20260620 18:33:12.364847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16099
Jun 20 18:33:12 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:33:12.379865 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.365583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.97.2:42637)
0620 18:33:12.365770 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.365775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.365792 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:33:12.365937 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.365941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.366015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.366100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.367337 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.368093 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.369582 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.369946 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.370035 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.374487 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:12.374502 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:12.374518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:12.374551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:12.377261 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.377858 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.377864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.377867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.377992 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.378370 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.378374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.378376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.378771 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:12.378993 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.379289 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.379449 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:33:12.380473 16222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:12.471066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16099
I20260620 18:33:12.487950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16233
Jun 20 18:33:12 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:33:12.503717 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.488701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.97.3:36129)
0620 18:33:12.488911 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.488918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.488936 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:33:12.489116 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.489122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.489208 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.489320 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.490717 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.491442 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.492765 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.493157 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.493292 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.497547 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:12.497562 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:12.497578 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:12.497618 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:12.500732 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.501475 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.501482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.501484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.501684 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:12.502009 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:12.502012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:12.502014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:12.502345 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:12.502555 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:12.502879 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:33:12.503030 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:33:12.504396 16356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:12.594403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16233
I20260620 18:33:12.612937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 15872
I20260620 18:33:12.614075 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:12.740088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 15872
I20260620 18:33:32.763944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--webserver_interface=127.0.97.62
--webserver_port=46071
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:33.123159 16410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:33.123431 16410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:33.123567 16410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:33.130618 16410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:33:33.130705 16410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:33:33.130744 16410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:33:33.130797 16410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:33.130851 16410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:33.130918 16410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:33.130962 16410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:33.131024 16410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:33.140673 16410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:33.141812 16410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:33.143087 16410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:33.165397 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:33.165362 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:33.165444 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:33.166538 16410 server_base.cc:1034] running on GCE node
I20260620 18:33:33.167466 16410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:33.169449 16410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:33.170714 16410 hybrid_clock.cc:648] HybridClock initialized: now 1781980413170625 us; error 59 us; skew 500 ppm
I20260620 18:33:33.171108 16410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:33 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:33.174747 16410 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:33:33.176513 16410 webserver.cc:460] Webserver started at http://127.0.97.62:46071/ using document root <none> and password file <none>
I20260620 18:33:33.177255 16410 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:33.177358 16410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:33.183419 16410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:33:33.186391 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:33.187942 16410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:33:33.188134 16410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "9383b3527f5e4ac0a853303dcd1a3a7b"
format_stamp: "Formatted at 2026-06-20 18:33:09 on dist-test-slave-mcnm"
server_key: "b59f2679bc4d0f0ed86383775329329e"
server_key_iv: "a2fd45fd329d34616ca361565e855849"
server_key_version: "encryptionkey@0"
I20260620 18:33:33.189281 16410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:33.202394 16410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:33.206888 16410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:33.207186 16410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:33.225909 16410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44853
I20260620 18:33:33.225943 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44853 every 8 connection(s)
I20260620 18:33:33.227447 16410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:33.229611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16410
I20260620 18:33:33.230677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40205
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=40273
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:33:33.237358 16478 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:33:33.241597 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap starting.
I20260620 18:33:33.261345 16478 log.cc:824] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:33.273029 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:33:33.273589 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap complete.
I20260620 18:33:33.282734 16478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:33.283612 16478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Initialized, Role: FOLLOWER
I20260620 18:33:33.284214 16478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:33.284446 16478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:33.284585 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:33.284765 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:33:33.288550 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:33.289060 16478 leader_election.cc:302] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9383b3527f5e4ac0a853303dcd1a3a7b; no voters:
I20260620 18:33:33.289709 16478 leader_election.cc:288] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:33:33.289840 16482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:33:33.291196 16482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 LEADER]: Becoming Leader. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Running, Role: LEADER
I20260620 18:33:33.291787 16482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:33.292143 16478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:33.294291 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9383b3527f5e4ac0a853303dcd1a3a7b. Latest consensus state: current_term: 2 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:33.294631 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:33.294380 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:33.294803 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:33.301301 16487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:33.311595 16487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:33.313496 16487 catalog_manager.cc:1259] Loaded cluster ID: d582f04d84a54c8b9960c5b3dca9d3df
I20260620 18:33:33.313704 16487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:33.319434 16487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:33.321617 16487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Loaded TSK: 0
I20260620 18:33:33.358016 16487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Generated new TSK 1
I20260620 18:33:33.358412 16487 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:33:33.598101 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:33.598390 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:33.598534 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:33.606030 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:33.606137 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:33.606293 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:33:33.616331 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40273
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:33.617565 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:33.618896 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:33.633612 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:33.633562 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:33.635808 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:33.637530 16480 server_base.cc:1034] running on GCE node
I20260620 18:33:33.638161 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:33.639151 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:33.640374 16480 hybrid_clock.cc:648] HybridClock initialized: now 1781980413640297 us; error 73 us; skew 500 ppm
I20260620 18:33:33.640694 16480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:33 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:33.643991 16480 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:33:33.645728 16480 webserver.cc:460] Webserver started at http://127.0.97.1:40273/ using document root <none> and password file <none>
I20260620 18:33:33.646467 16480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:33.646605 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:33.652073 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:33.654775 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:33.656309 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:33.656502 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "5e85526cfdf849758c4b115a9bfd0cde"
format_stamp: "Formatted at 2026-06-20 18:33:10 on dist-test-slave-mcnm"
server_key: "1dae4f0f4c5135e8f5235fbcb7594242"
server_key_iv: "83a40d7210079b663b537fcce26c924e"
server_key_version: "encryptionkey@0"
I20260620 18:33:33.657544 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:33.700068 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:33.704893 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:33.705224 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:33.706420 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:33.708057 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:33.708186 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:33.708329 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:33.708424 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:33.744343 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40205
I20260620 18:33:33.744402 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40205 every 8 connection(s)
I20260620 18:33:33.746019 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:33.747396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16480
I20260620 18:33:33.748103 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42637
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=45629
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:33:33 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:33.773947 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:33.749794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:52517)
0620 18:33:33.750595 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:33.750637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:33.752438 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:33.753866 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:33.753878 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:33.754508 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:33.755042 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:33.755060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:33.757258 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:33.757267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:33.759545 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:33.759564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:33.761157 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:33.761167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:33.761481 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:33.762280 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:33.762320 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:33.765058 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:33.769220 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:33.769231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:33.769248 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:33.769735 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:33.770221 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:33.770226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:33.770231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:33.770440 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:33.771227 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:33.771236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:33.771655 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 18:33:33.771928 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 18:33:33.772536 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":317,"threads_started":1}
I20260620 18:33:33.773965 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:33.749382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52517 (local address 127.0.97.62:44853)
0620 18:33:33.750002 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:33.750015 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:33.750777 (+ 762us) server_negotiation.cc:407] Connection header received
0620 18:33:33.753008 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:33.753021 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:33.753353 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:33.753720 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:33.755242 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:33.757086 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:33.759717 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:33.760882 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:33.761064 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:33.765368 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:33.765402 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:33.765420 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:33.765460 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:33.769064 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:33.769898 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:33.769908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:33.769916 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:33.770097 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:33.770565 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:33.770570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:33.770574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:33.771082 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:33.771292 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:33.772178 (+ 886us) server_negotiation.cc:299] Negotiation successful
0620 18:33:33.772575 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
I20260620 18:33:33.777071 16628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:33.777415 16628 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:33.778262 16628 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
I20260620 18:33:33.780586 16443 ts_manager.cc:194] Registered new tserver with Master: 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205)
I20260620 18:33:33.782130 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52517
W20260620 18:33:34.084770 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:34.085068 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:34.085214 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:34.092670 16632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:34.092782 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:34.093012 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:33:34.103080 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45629
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:34.104413 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:34.105751 16632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:34.118894 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:34.119069 16640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:34.119199 16638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:34.120540 16632 server_base.cc:1034] running on GCE node
I20260620 18:33:34.121202 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:34.122171 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:34.123443 16632 hybrid_clock.cc:648] HybridClock initialized: now 1781980414123381 us; error 67 us; skew 500 ppm
I20260620 18:33:34.123857 16632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:34 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:34.127202 16632 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:33:34.128816 16632 webserver.cc:460] Webserver started at http://127.0.97.2:45629/ using document root <none> and password file <none>
I20260620 18:33:34.129606 16632 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:34.129770 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:34.135555 16632 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:33:34.138737 16647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.140300 16632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:34.140480 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "2f46527d9ad44093b016a0c5100c8b1f"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "fcf02f4daecb4f31a0f5ac766d0da97f"
server_key_iv: "cd5b0e796d3618c412076b41b7831c92"
server_key_version: "encryptionkey@0"
I20260620 18:33:34.141495 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:34.158761 16632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:34.163065 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:34.163354 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:34.164373 16632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:34.165748 16632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:34.165833 16632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.165971 16632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:34.166060 16632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.197690 16632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42637
I20260620 18:33:34.197737 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42637 every 8 connection(s)
I20260620 18:33:34.199198 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:34.205910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16632
I20260620 18:33:34.206681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36129
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=36559
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:33:34 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:34.222728 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.201768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41689 (local address 127.0.97.62:44853)
0620 18:33:34.201945 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.201949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.203129 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 18:33:34.205040 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.205046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.205135 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.205248 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:34.207298 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.208066 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.210610 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.211183 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.211319 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.215662 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:34.215672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:34.215678 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:34.215725 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:34.219347 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.220123 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.220131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.220137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.220305 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.220754 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.220761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.220767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.221149 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:34.221293 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.222009 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.222285 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:33:34.223454 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.202189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:41689)
0620 18:33:34.202938 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:34.202987 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:34.204767 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:34.205525 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:34.205540 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:34.206292 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:34.207112 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.207140 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.208231 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.208238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.210405 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.210422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.211435 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.211445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.211830 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.212584 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:34.212618 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:34.215398 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:34.219557 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:34.219572 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:34.219586 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:34.219998 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:34.220408 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:34.220412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:34.220414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:34.220632 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:34.221292 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:34.221298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:34.221644 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 18:33:34.221874 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 18:33:34.222250 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":324,"threads_started":1}
I20260620 18:33:34.224967 16760 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:34.225289 16760 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:34.226033 16760 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
I20260620 18:33:34.227828 16443 ts_manager.cc:194] Registered new tserver with Master: 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:34.228647 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41689
W20260620 18:33:34.577720 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:34.578056 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:34.578217 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:34.585632 16763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:34.585789 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:34.585964 16763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:33:34.596469 16763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36559
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:34.597934 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:34.599324 16763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:34.612722 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:34.612783 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:34.612766 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:34.613533 16763 server_base.cc:1034] running on GCE node
I20260620 18:33:34.614179 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:34.615131 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:34.616380 16763 hybrid_clock.cc:648] HybridClock initialized: now 1781980414616324 us; error 57 us; skew 500 ppm
I20260620 18:33:34.616760 16763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:34 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:34.620050 16763 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:33:34.621644 16763 webserver.cc:460] Webserver started at http://127.0.97.3:36559/ using document root <none> and password file <none>
I20260620 18:33:34.622311 16763 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:34.622430 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:34.628101 16763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:33:34.630807 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.632266 16763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:34.632433 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "5fa08d45f83248c39105a008fbc4e90a"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "33ea8baa94ed48e34468a2c65fb112d3"
server_key_iv: "09c0d3ce774d63059748cf2d438d8f56"
server_key_version: "encryptionkey@0"
I20260620 18:33:34.633440 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:34.654852 16763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:34.658866 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:34.659129 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:34.660152 16763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:34.661573 16763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:34.661656 16763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.661832 16763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:34.661924 16763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:34.696626 16763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36129
I20260620 18:33:34.696713 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36129 every 8 connection(s)
I20260620 18:33:34.698213 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:34.705492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16763
Jun 20 18:33:34 dist-test-slave-mcnm krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:33:34.721640 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.700961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52665 (local address 127.0.97.62:44853)
0620 18:33:34.701153 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.701158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.702428 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 18:33:34.704471 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.704475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.704546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.704650 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:34.706717 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.707933 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.710146 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.710806 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.710905 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.715242 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:34.715246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:34.715249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:34.715292 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:34.718463 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.719298 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.719306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.719313 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.719461 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.719910 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.719916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.719920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.720212 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:34.720354 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.721034 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.721242 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":5504}
I20260620 18:33:34.722616 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.701477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:52665)
0620 18:33:34.702233 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:34.702276 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:34.704184 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:34.704902 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:34.704911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:34.705734 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:34.706502 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.706527 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.708068 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.708076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.710017 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.710028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.711066 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.711073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.711353 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.712134 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:34.712172 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:34.715023 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:34.718604 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:34.718614 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:34.718630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:34.719173 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:34.719612 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:34.719618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:34.719621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:34.719801 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:34.720354 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:34.720362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:34.720683 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 18:33:34.720945 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:33:34.721494 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":305,"threads_started":1}
I20260620 18:33:34.722924 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.706507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.0.97.62:44853)
0620 18:33:34.707397 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.707403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.707420 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:33:34.707465 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.707469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.707569 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.707678 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:34.709717 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.710695 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.712616 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.713069 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.713178 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.716451 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:34.716457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:34.716462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:34.716530 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:34.720091 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.720719 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.720726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.720729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.720856 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.721264 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.721268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.721270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.721527 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:34.721809 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.722303 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.722520 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"spinlock_wait_cycles":1408,"thread_start_us":647,"threads_started":1}
I20260620 18:33:34.724241 16891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44853
I20260620 18:33:34.724640 16891 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:34.725528 16891 heartbeater.cc:511] Master 127.0.97.62:44853 requested a full tablet report, sending...
I20260620 18:33:34.726984 16442 ts_manager.cc:194] Registered new tserver with Master: 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129)
I20260620 18:33:34.728003 16442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52665
I20260620 18:33:34.737948 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:33:34.745501 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.738990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.97.62:44853)
0620 18:33:34.739178 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.739183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.739714 (+ 531us) server_negotiation.cc:407] Connection header received
0620 18:33:34.739960 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.739964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.740102 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.740226 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:33:34.740789 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.741547 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.743173 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.743743 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.743874 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.744507 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:33:34.744763 (+ 256us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:33:34.745189 (+ 426us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39312: token expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 18:33:34.745637 16895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39312: token expired
W20260620 18:33:34.746182 16367 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:44853)
I20260620 18:33:34.759325 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.746684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.0.97.62:44853)
0620 18:33:34.746838 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.746845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.746866 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:33:34.747189 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.747194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.747265 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.747367 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:34.748939 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.749698 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.751111 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.751503 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.751681 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.754264 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:34.754269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:34.754274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:34.754317 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:34.756720 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.757394 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.757399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.757403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.757536 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:34.757888 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:34.757894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:34.757898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:34.758213 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:34.758455 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.758765 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.758999 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:33:34.763552 16442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:33:34.773005 16442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:33:34.785789 16628 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
I20260620 18:33:34.807844 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.785282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.0.97.3:36129)
0620 18:33:34.797783 (+ 12501us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.797796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.797837 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:33:34.797957 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.797968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.798269 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.798484 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:34.800689 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.802402 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.805115 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.806620 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.806760 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.807001 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.807302 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.807475 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12270,"thread_start_us":12148,"threads_started":1}
I20260620 18:33:34.807874 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.784827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36129 (local address 127.0.0.1:41698)
0620 18:33:34.796733 (+ 11906us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:34.796813 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:34.797035 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:34.800007 (+ 2972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:34.800016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:34.800048 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:34.800506 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.800518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.802576 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.802584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.804963 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.804978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.806838 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.806844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.807030 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.807084 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:33:34.807214 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:33:34.807421 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11609,"mutex_wait_us":43,"thread_start_us":13890,"threads_started":1}
I20260620 18:33:34.810053 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.784998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40205 (local address 127.0.0.1:34696)
0620 18:33:34.798283 (+ 13285us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:34.798308 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:34.798469 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:34.801680 (+ 3211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:34.801685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:34.801711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:34.802085 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.802096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.804419 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.804423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.806406 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.806420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.809170 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.809176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.809352 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.809382 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:33:34.809501 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:33:34.809644 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13073,"mutex_wait_us":4,"thread_start_us":11272,"threads_started":1}
I20260620 18:33:34.810072 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.785352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.0.97.1:40205)
0620 18:33:34.801136 (+ 15784us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.801143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.801171 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:33:34.801256 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.801263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.801482 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.801672 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:34.802466 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.804296 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.806591 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.808946 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.809081 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.809300 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.809547 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.809714 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15613,"thread_start_us":15444,"threads_started":1}
I20260620 18:33:34.810834 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.788582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.97.2:42637)
0620 18:33:34.799243 (+ 10661us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:34.799256 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:34.800172 (+ 916us) server_negotiation.cc:407] Connection header received
0620 18:33:34.800298 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:34.800311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:34.800621 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:34.800839 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:34.801485 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.802952 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.807368 (+ 4416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.809078 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.809209 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.809396 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:34.810455 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0620 18:33:34.810581 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10455,"thread_start_us":10317,"threads_started":1}
I20260620 18:33:34.811904 16826 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab (DEFAULT_TABLE table=test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:34.813194 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb66719c8247bfacc14bbfcc5c33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:34.813438 16563 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab (DEFAULT_TABLE table=test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:34.814576 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb66719c8247bfacc14bbfcc5c33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:34.817030 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:34.787856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42637 (local address 127.0.0.1:56684)
0620 18:33:34.799831 (+ 11975us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:34.799859 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:34.800018 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:34.801078 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:34.801081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:34.801100 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:34.801324 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.801331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.803080 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.803087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.807199 (+ 4112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:34.807213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:34.809566 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:34.809572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:34.809722 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:34.809774 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:33:34.809859 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:33:34.809985 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11796,"mutex_wait_us":2362,"thread_start_us":12416,"threads_started":1}
I20260620 18:33:34.820467 16695 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab (DEFAULT_TABLE table=test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:34.821863 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb66719c8247bfacc14bbfcc5c33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:34.825237 16907 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Bootstrap starting.
I20260620 18:33:34.828460 16907 tablet_bootstrap.cc:654] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:34.829577 16907 log.cc:824] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:34.830143 16908 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Bootstrap starting.
I20260620 18:33:34.833729 16908 tablet_bootstrap.cc:654] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:34.834051 16907 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: No bootstrap required, opened a new log
I20260620 18:33:34.834343 16907 ts_tablet_manager.cc:1397] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 18:33:34.834872 16908 log.cc:824] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:34.837818 16908 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: No bootstrap required, opened a new log
I20260620 18:33:34.838102 16908 ts_tablet_manager.cc:1397] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:33:34.838150 16911 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Bootstrap starting.
I20260620 18:33:34.841941 16911 tablet_bootstrap.cc:654] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:34.843230 16911 log.cc:824] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:34.843736 16907 raft_consensus.cc:348] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.844293 16907 raft_consensus.cc:374] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:34.844426 16907 raft_consensus.cc:729] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa08d45f83248c39105a008fbc4e90a, State: Initialized, Role: FOLLOWER
I20260620 18:33:34.845193 16907 consensus_queue.cc:260] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.846654 16911 tablet_bootstrap.cc:492] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: No bootstrap required, opened a new log
I20260620 18:33:34.846621 16891 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
I20260620 18:33:34.846928 16911 ts_tablet_manager.cc:1397] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 18:33:34.847091 16907 ts_tablet_manager.cc:1428] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:33:34.849817 16908 raft_consensus.cc:348] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.850287 16908 raft_consensus.cc:374] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:34.850382 16908 raft_consensus.cc:729] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e85526cfdf849758c4b115a9bfd0cde, State: Initialized, Role: FOLLOWER
I20260620 18:33:34.851087 16908 consensus_queue.cc:260] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.852842 16908 ts_tablet_manager.cc:1428] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260620 18:33:34.856338 16911 raft_consensus.cc:348] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.856712 16911 raft_consensus.cc:374] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:34.856817 16911 raft_consensus.cc:729] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f46527d9ad44093b016a0c5100c8b1f, State: Initialized, Role: FOLLOWER
I20260620 18:33:34.857316 16911 consensus_queue.cc:260] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:34.858289 16760 heartbeater.cc:503] Master 127.0.97.62:44853 was elected leader, sending a full tablet report...
I20260620 18:33:34.858692 16911 ts_tablet_manager.cc:1428] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
W20260620 18:33:34.949985 16892 tablet.cc:2369] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:33:34.951126 16761 tablet.cc:2369] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:33:34.998596 16629 tablet.cc:2369] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:33:35.099416 16913 raft_consensus.cc:482] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:33:35.099790 16913 raft_consensus.cc:504] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:35.101552 16913 leader_election.cc:288] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205), 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:35.107097 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.101963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55003 (local address 127.0.97.2:42637)
0620 18:33:35.102186 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.102192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.102337 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:33:35.102564 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.102568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.102628 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.102723 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:35.103098 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.103870 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.105433 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.106201 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.106345 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.106435 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.106708 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.106836 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:33:35.107116 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.101835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42637 (local address 127.0.97.3:55003)
0620 18:33:35.102236 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:35.102259 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:35.102411 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:35.102744 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:35.102749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:35.102780 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:35.102957 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:35.102965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.104009 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.104016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:35.105278 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:35.105289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.106357 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.106363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:35.106554 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.106578 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:33:35.106681 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:33:35.106808 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260620 18:33:35.107926 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.101945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40205 (local address 127.0.97.3:45059)
0620 18:33:35.102782 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:35.102796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:35.102950 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:35.103265 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:35.103269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:35.103291 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:35.103676 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:35.103685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.104670 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.104676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:35.106014 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:35.106025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.107160 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.107166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:35.107372 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.107410 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:33:35.107505 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:33:35.107663 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":548,"threads_started":1}
I20260620 18:33:35.107951 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.101988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45059 (local address 127.0.97.1:40205)
0620 18:33:35.102303 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.102312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.102895 (+ 583us) server_negotiation.cc:407] Connection header received
0620 18:33:35.103094 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.103098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.103157 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.103251 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:35.103822 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.104526 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.106179 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.106949 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.107103 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.107207 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.107574 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.107682 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 18:33:35.108290 16715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb66719c8247bfacc14bbfcc5c33ab" candidate_uuid: "5fa08d45f83248c39105a008fbc4e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f46527d9ad44093b016a0c5100c8b1f" is_pre_election: true
I20260620 18:33:35.108862 16715 raft_consensus.cc:2454] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa08d45f83248c39105a008fbc4e90a in term 0.
I20260620 18:33:35.109089 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb66719c8247bfacc14bbfcc5c33ab" candidate_uuid: "5fa08d45f83248c39105a008fbc4e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e85526cfdf849758c4b115a9bfd0cde" is_pre_election: true
I20260620 18:33:35.109580 16782 leader_election.cc:302] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f46527d9ad44093b016a0c5100c8b1f, 5fa08d45f83248c39105a008fbc4e90a; no voters:
I20260620 18:33:35.109624 16583 raft_consensus.cc:2454] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa08d45f83248c39105a008fbc4e90a in term 0.
I20260620 18:33:35.109974 16913 raft_consensus.cc:2781] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:33:35.110150 16913 raft_consensus.cc:482] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:33:35.110272 16913 raft_consensus.cc:3037] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:35.112512 16913 raft_consensus.cc:504] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:35.113121 16913 leader_election.cc:288] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [CANDIDATE]: Term 1 election: Requested vote from peers 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205), 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637)
I20260620 18:33:35.113538 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb66719c8247bfacc14bbfcc5c33ab" candidate_uuid: "5fa08d45f83248c39105a008fbc4e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e85526cfdf849758c4b115a9bfd0cde"
I20260620 18:33:35.113610 16715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb66719c8247bfacc14bbfcc5c33ab" candidate_uuid: "5fa08d45f83248c39105a008fbc4e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f46527d9ad44093b016a0c5100c8b1f"
I20260620 18:33:35.113756 16583 raft_consensus.cc:3037] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:35.113783 16715 raft_consensus.cc:3037] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:35.117149 16583 raft_consensus.cc:2454] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa08d45f83248c39105a008fbc4e90a in term 1.
I20260620 18:33:35.117149 16715 raft_consensus.cc:2454] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa08d45f83248c39105a008fbc4e90a in term 1.
I20260620 18:33:35.117564 16781 leader_election.cc:302] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e85526cfdf849758c4b115a9bfd0cde, 5fa08d45f83248c39105a008fbc4e90a; no voters:
I20260620 18:33:35.117910 16913 raft_consensus.cc:2781] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:35.118397 16913 raft_consensus.cc:686] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 1 LEADER]: Becoming Leader. State: Replica: 5fa08d45f83248c39105a008fbc4e90a, State: Running, Role: LEADER
I20260620 18:33:35.118920 16913 consensus_queue.cc:237] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } }
I20260620 18:33:35.123957 16443 catalog_manager.cc:5495] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5fa08d45f83248c39105a008fbc4e90a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa08d45f83248c39105a008fbc4e90a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260620 18:33:35.178648 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.173110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.97.3:36129)
0620 18:33:35.173309 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.173316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.173416 (+ 100us) server_negotiation.cc:407] Connection header received
0620 18:33:35.173677 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.173682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.173760 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.173916 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:33:35.174307 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.175030 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.176501 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.176906 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.177034 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.177572 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:33:35.178081 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:33:35.178245 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.178296 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.178376 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:33:35.187407 16583 raft_consensus.cc:1261] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Refusing update from remote peer 5fa08d45f83248c39105a008fbc4e90a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:35.187417 16715 raft_consensus.cc:1261] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Refusing update from remote peer 5fa08d45f83248c39105a008fbc4e90a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:35.188393 16917 consensus_queue.cc:1035] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f46527d9ad44093b016a0c5100c8b1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:35.188759 16913 consensus_queue.cc:1035] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e85526cfdf849758c4b115a9bfd0cde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:35.204484 16920 mvcc.cc:205] Tried to move back new op lower bound from 7298991780596285440 to 7298991780330409984. Current Snapshot: MvccSnapshot[applied={T|T < 7298991780596285440}]
I20260620 18:33:35.206295 16922 mvcc.cc:205] Tried to move back new op lower bound from 7298991780596285440 to 7298991780330409984. Current Snapshot: MvccSnapshot[applied={T|T < 7298991780596285440}]
I20260620 18:33:35.212879 16921 mvcc.cc:205] Tried to move back new op lower bound from 7298991780596285440 to 7298991780330409984. Current Snapshot: MvccSnapshot[applied={T|T < 7298991780596285440}]
I20260620 18:33:35.225903 16443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:35.226200 16443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:35.230129 16443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Sending DeleteTablet for 3 replicas of tablet a6eb66719c8247bfacc14bbfcc5c33ab
I20260620 18:33:35.231267 16826 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41698
I20260620 18:33:35.231312 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34696
I20260620 18:33:35.231844 16695 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eb66719c8247bfacc14bbfcc5c33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56684
I20260620 18:33:35.233309 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16480
I20260620 18:33:35.244158 16929 tablet_replica.cc:330] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: stopping tablet replica
I20260620 18:33:35.244701 16929 raft_consensus.cc:2229] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:35.245033 16930 tablet_replica.cc:330] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: stopping tablet replica
I20260620 18:33:35.245414 16929 raft_consensus.cc:2258] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:35.247877 16929 ts_tablet_manager.cc:1905] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:35.248453 16931 tablet_replica.cc:330] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: stopping tablet replica
I20260620 18:33:35.249123 16931 raft_consensus.cc:2229] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:35.250838 16931 raft_consensus.cc:2258] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:35.253587 16931 ts_tablet_manager.cc:1905] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:35.261051 16931 ts_tablet_manager.cc:1918] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:35.261225 16931 log.cc:1196] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/a6eb66719c8247bfacc14bbfcc5c33ab
I20260620 18:33:35.261772 16931 ts_tablet_manager.cc:1939] T a6eb66719c8247bfacc14bbfcc5c33ab P 2f46527d9ad44093b016a0c5100c8b1f: Deleting consensus metadata
I20260620 18:33:35.263553 16930 raft_consensus.cc:2229] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:33:35.263911 16430 catalog_manager.cc:4841] TS 2f46527d9ad44093b016a0c5100c8b1f (127.0.97.2:42637): tablet a6eb66719c8247bfacc14bbfcc5c33ab (table test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]) successfully deleted
I20260620 18:33:35.264252 16930 raft_consensus.cc:2258] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:35.266474 16929 ts_tablet_manager.cc:1918] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:35.266667 16929 log.cc:1196] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/a6eb66719c8247bfacc14bbfcc5c33ab
I20260620 18:33:35.267158 16929 ts_tablet_manager.cc:1939] T a6eb66719c8247bfacc14bbfcc5c33ab P 5e85526cfdf849758c4b115a9bfd0cde: Deleting consensus metadata
I20260620 18:33:35.268970 16429 catalog_manager.cc:4841] TS 5e85526cfdf849758c4b115a9bfd0cde (127.0.97.1:40205): tablet a6eb66719c8247bfacc14bbfcc5c33ab (table test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]) successfully deleted
I20260620 18:33:35.269748 16930 ts_tablet_manager.cc:1905] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:35.271404 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.234722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.0.97.1:40205)
0620 18:33:35.234906 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.234913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.250035 (+ 15122us) server_negotiation.cc:407] Connection header received
0620 18:33:35.250269 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.250275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.250360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.250463 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:35.252831 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.254068 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.256379 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.257156 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.257257 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.261214 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:35.261230 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:35.261254 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:35.261311 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:35.267768 (+ 6457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.268715 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.268726 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.268730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.268908 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.269447 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.269453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.269457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.269922 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:35.270211 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.270797 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.271017 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:33:35.272939 16623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:35.276788 16930 ts_tablet_manager.cc:1918] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:35.276955 16930 log.cc:1196] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/a6eb66719c8247bfacc14bbfcc5c33ab
I20260620 18:33:35.277438 16930 ts_tablet_manager.cc:1939] T a6eb66719c8247bfacc14bbfcc5c33ab P 5fa08d45f83248c39105a008fbc4e90a: Deleting consensus metadata
I20260620 18:33:35.278831 16427 catalog_manager.cc:4841] TS 5fa08d45f83248c39105a008fbc4e90a (127.0.97.3:36129): tablet a6eb66719c8247bfacc14bbfcc5c33ab (table test-table [id=18f6dc33ffe04200b909f25a8b1f4bd8]) successfully deleted
I20260620 18:33:35.409586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16480
I20260620 18:33:35.427136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16632
I20260620 18:33:35.442401 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.428017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.0.97.2:42637)
0620 18:33:35.428187 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.428192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.428244 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:33:35.428468 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.428472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.428539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.428624 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:35.430332 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.431028 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.432976 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.433357 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.433450 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.436807 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:35.436821 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:35.436836 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:35.436867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:35.439741 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.440569 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.440575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.440578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.440714 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.441094 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.441097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.441099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.441438 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:35.441643 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.441967 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.442119 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:33:35.443032 16755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:35.532608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16632
I20260620 18:33:35.549630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16763
I20260620 18:33:35.567500 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:35.550408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.97.3:36129)
0620 18:33:35.550586 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:35.550595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:35.550674 (+ 79us) server_negotiation.cc:407] Connection header received
0620 18:33:35.550895 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:35.550899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:35.550967 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:35.551075 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:35.552416 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.553246 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.554538 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:35.555005 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:35.555145 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:35.561259 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:35.561272 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:35.561287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:35.561331 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:35.564688 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.565401 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.565414 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.565416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.565574 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:35.565976 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:35.565981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:35.565989 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:35.566349 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:35.566642 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:35.566930 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:33:35.567158 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:33:35.568222 16886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:35.689615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16763
I20260620 18:33:35.709949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 16410
I20260620 18:33:35.711074 16473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:35.825774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 16410
I20260620 18:33:55.848249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--webserver_interface=127.0.97.62
--webserver_port=46071
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:56.192682 16937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:56.193029 16937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:56.193205 16937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:56.200485 16937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:33:56.200639 16937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:33:56.200706 16937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:33:56.200789 16937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:56.200886 16937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:56.201025 16937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:56.201123 16937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:56.201205 16937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:56.211078 16937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:56.212415 16937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:56.213792 16937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:56.227241 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:56.227210 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:56.227293 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:56.227636 16937 server_base.cc:1034] running on GCE node
I20260620 18:33:56.228497 16937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:56.229818 16937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:56.231068 16937 hybrid_clock.cc:648] HybridClock initialized: now 1781980436230993 us; error 54 us; skew 500 ppm
I20260620 18:33:56.231397 16937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:56 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:56.234668 16937 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:33:56.236114 16937 webserver.cc:460] Webserver started at http://127.0.97.62:46071/ using document root <none> and password file <none>
I20260620 18:33:56.236752 16937 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:56.236835 16937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:56.242089 16937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:33:56.244844 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:56.246276 16937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:56.246477 16937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "9383b3527f5e4ac0a853303dcd1a3a7b"
format_stamp: "Formatted at 2026-06-20 18:33:09 on dist-test-slave-mcnm"
server_key: "b59f2679bc4d0f0ed86383775329329e"
server_key_iv: "a2fd45fd329d34616ca361565e855849"
server_key_version: "encryptionkey@0"
I20260620 18:33:56.247414 16937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:56.264814 16937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:56.269778 16937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:56.270081 16937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:56.289286 16937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44853
I20260620 18:33:56.289373 17003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44853 every 8 connection(s)
I20260620 18:33:56.290784 16937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:56.292917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 16937
I20260620 18:33:56.293989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40205
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=40273
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:33:56.300400 17004 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:33:56.304790 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap starting.
I20260620 18:33:56.325769 17004 log.cc:824] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:56.341567 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:33:56.342177 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Bootstrap complete.
I20260620 18:33:56.352165 17004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:56.353123 17004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Initialized, Role: FOLLOWER
I20260620 18:33:56.353760 17004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:56.353996 17004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:56.354102 17004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:56.354264 17004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:33:56.357942 17004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:56.358404 17004 leader_election.cc:302] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9383b3527f5e4ac0a853303dcd1a3a7b; no voters:
I20260620 18:33:56.359099 17004 leader_election.cc:288] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:33:56.359236 17008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:33:56.360476 17008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [term 3 LEADER]: Becoming Leader. State: Replica: 9383b3527f5e4ac0a853303dcd1a3a7b, State: Running, Role: LEADER
I20260620 18:33:56.361029 17008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } }
I20260620 18:33:56.361307 17004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:56.363682 17009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:56.364037 17009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:56.363408 17010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9383b3527f5e4ac0a853303dcd1a3a7b. Latest consensus state: current_term: 3 leader_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9383b3527f5e4ac0a853303dcd1a3a7b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44853 } } }
I20260620 18:33:56.364777 17010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:56.376567 17015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:56.386075 17015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:56.387490 17015 catalog_manager.cc:1259] Loaded cluster ID: d582f04d84a54c8b9960c5b3dca9d3df
I20260620 18:33:56.387604 17015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:56.391496 17015 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:56.393199 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Loaded TSK: 0
I20260620 18:33:56.393344 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9383b3527f5e4ac0a853303dcd1a3a7b: Loaded TSK: 1
F20260620 18:33:56.394381 17015 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5c959f29ad google::LogMessage::Fail() at ??:0
@ 0x7f5c959f588a google::LogMessage::SendToLog() at ??:0
@ 0x7f5c959f24de google::LogMessage::Flush() at ??:0
@ 0x7f5c959f3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5c997567d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5c9974573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5cab4211c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5cab420783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5cab4ac2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5cab4ac0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5cab4f65e9 std::function<>::operator()() at ??:0
@ 0x7f5cab424cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5cab422c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5cab4aa610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5cab4aa439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5c9863e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5c9864191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5c98641799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5c9861ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5c9b8f66db start_thread at ??:0
@ 0x7f5c9488c61f clone at ??:0
W20260620 18:33:56.675053 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:56.675376 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:56.675604 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:56.682907 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:56.683068 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:56.683256 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:33:56.693081 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40273
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:56.694391 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:56.695757 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:56.713598 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:56.714402 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:56.714439 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:56.716346 17006 server_base.cc:1034] running on GCE node
I20260620 18:33:56.717073 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:56.718070 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:56.719298 17006 hybrid_clock.cc:648] HybridClock initialized: now 1781980436719237 us; error 41 us; skew 500 ppm
I20260620 18:33:56.719718 17006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:56 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:56.722993 17006 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:33:56.724607 17006 webserver.cc:460] Webserver started at http://127.0.97.1:40273/ using document root <none> and password file <none>
I20260620 18:33:56.725350 17006 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:56.725502 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:56.730975 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:33:56.733868 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:56.735250 17006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:33:56.735432 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "5e85526cfdf849758c4b115a9bfd0cde"
format_stamp: "Formatted at 2026-06-20 18:33:10 on dist-test-slave-mcnm"
server_key: "1dae4f0f4c5135e8f5235fbcb7594242"
server_key_iv: "83a40d7210079b663b537fcce26c924e"
server_key_version: "encryptionkey@0"
I20260620 18:33:56.736510 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:56.752943 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:56.757170 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:56.757479 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:56.759328 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:56.761492 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:56.761657 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:56.761778 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:56.761878 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:56.800925 17006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40205
I20260620 18:33:56.800988 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40205 every 8 connection(s)
I20260620 18:33:56.802430 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:56.807274 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:56.805433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:43717)
0620 18:33:56.806220 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:56.806411 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":324,"threads_started":1}
W20260620 18:33:56.808035 17174 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44853 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44853: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
I20260620 18:33:56.808903 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:56.808546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:43273)
0620 18:33:56.808726 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:56.808803 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:33:56.809697 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:56.809398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:54537)
0620 18:33:56.809513 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:56.809589 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:33:56.809942 17174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:33:56.811165 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17006
I20260620 18:33:56.811818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42637
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=45629
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:57.143255 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:57.143651 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:57.143821 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:57.151105 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:57.151207 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:57.151381 17177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:33:57.160873 17177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45629
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:57.162137 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:57.163455 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:57.176782 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:57.176893 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:57.177371 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:57.177924 17177 server_base.cc:1034] running on GCE node
I20260620 18:33:57.178596 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:57.179654 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:57.180902 17177 hybrid_clock.cc:648] HybridClock initialized: now 1781980437180848 us; error 42 us; skew 500 ppm
I20260620 18:33:57.181233 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:57 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:57.184438 17177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:33:57.185883 17177 webserver.cc:460] Webserver started at http://127.0.97.2:45629/ using document root <none> and password file <none>
I20260620 18:33:57.186579 17177 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:57.186733 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:57.192061 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:33:57.194773 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.196200 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:57.196375 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "2f46527d9ad44093b016a0c5100c8b1f"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "fcf02f4daecb4f31a0f5ac766d0da97f"
server_key_iv: "cd5b0e796d3618c412076b41b7831c92"
server_key_version: "encryptionkey@0"
I20260620 18:33:57.197366 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:57.211408 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:57.215416 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:57.215730 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:57.216707 17177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:57.218014 17177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:57.218142 17177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.218279 17177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:57.218369 17177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.251950 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42637
I20260620 18:33:57.252005 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42637 every 8 connection(s)
I20260620 18:33:57.253379 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:57.255566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17177
I20260620 18:33:57.256194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36129
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44853
--webserver_port=36559
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:33:57.258478 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.256477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:59609)
0620 18:33:57.257351 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.257593 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":360,"threads_started":1}
W20260620 18:33:57.259195 17305 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44853 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44853: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
I20260620 18:33:57.260114 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.259783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:49969)
0620 18:33:57.259944 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.260013 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:33:57.260991 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.260650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:54015)
0620 18:33:57.260802 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.260883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:33:57.261241 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:33:57.580471 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:57.580777 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:57.580938 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:57.588080 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:57.588179 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:57.588351 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:33:57.597865 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36559
--tserver_master_addrs=127.0.97.62:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:33:57.599102 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:57.600503 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:57.613572 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:57.613626 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:57.613581 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:57.614607 17308 server_base.cc:1034] running on GCE node
I20260620 18:33:57.615186 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:57.616171 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:57.617373 17308 hybrid_clock.cc:648] HybridClock initialized: now 1781980437617337 us; error 35 us; skew 500 ppm
I20260620 18:33:57.617684 17308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:57 dist-test-slave-mcnm krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:57.620970 17308 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:33:57.622388 17308 webserver.cc:460] Webserver started at http://127.0.97.3:36559/ using document root <none> and password file <none>
I20260620 18:33:57.623062 17308 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:57.623211 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:57.628410 17308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:33:57.631055 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.632468 17308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:33:57.632648 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "5fa08d45f83248c39105a008fbc4e90a"
format_stamp: "Formatted at 2026-06-20 18:33:11 on dist-test-slave-mcnm"
server_key: "33ea8baa94ed48e34468a2c65fb112d3"
server_key_iv: "09c0d3ce774d63059748cf2d438d8f56"
server_key_version: "encryptionkey@0"
I20260620 18:33:57.633580 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:57.650452 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:57.654222 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:57.654485 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:57.655402 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:57.656693 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:57.656831 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.656963 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:57.657056 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:57.691323 17308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36129
I20260620 18:33:57.691378 17435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36129 every 8 connection(s)
I20260620 18:33:57.692744 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:57.697602 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.695826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:34423)
0620 18:33:57.696539 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.696732 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":282,"threads_started":1}
W20260620 18:33:57.698261 17436 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44853 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44853: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
I20260620 18:33:57.699024 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.698716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:38551)
0620 18:33:57.698840 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.698927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:33:57.699764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17308
I20260620 18:33:57.699995 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.699603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:55715)
0620 18:33:57.699755 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.699852 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:33:57.700259 17436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:33:57.811371 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:57.810566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:46899)
0620 18:33:57.811051 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:57.811140 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260620 18:33:58.262537 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:58.261765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:36093)
0620 18:33:58.262203 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:58.262314 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260620 18:33:58.701996 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:58.700934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:45911)
0620 18:33:58.701595 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:58.701730 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":339,"threads_started":1}
I20260620 18:33:58.813174 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:58.812422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:38249)
0620 18:33:58.812908 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:58.812995 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260620 18:33:59.264151 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:59.263401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:42683)
0620 18:33:59.263903 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:59.263992 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":258,"threads_started":1}
I20260620 18:33:59.703712 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:59.702849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:47615)
0620 18:33:59.703360 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:59.703452 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
I20260620 18:33:59.814939 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:59.814205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:33239)
0620 18:33:59.814682 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:59.814763 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260620 18:34:00.265847 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:00.265160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:52751)
0620 18:34:00.265596 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:00.265681 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
I20260620 18:34:00.705396 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:00.704713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:42169)
0620 18:34:00.705154 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:00.705232 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260620 18:34:00.816692 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:00.815966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:50365)
0620 18:34:00.816441 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:00.816522 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260620 18:34:01.267431 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:01.266752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:56363)
0620 18:34:01.267167 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:01.267245 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260620 18:34:01.707096 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:01.706343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:47245)
0620 18:34:01.706831 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:01.706909 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":231,"threads_started":1}
I20260620 18:34:01.818409 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:01.817680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:39299)
0620 18:34:01.818161 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:01.818238 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260620 18:34:02.269199 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:02.268464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:46085)
0620 18:34:02.268944 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:02.269025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260620 18:34:02.708782 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:02.708078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:39665)
0620 18:34:02.708538 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:02.708617 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260620 18:34:02.820097 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:02.819361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:34741)
0620 18:34:02.819858 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:02.819941 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":290,"threads_started":1}
I20260620 18:34:03.271162 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:03.270213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:57943)
0620 18:34:03.270788 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:03.270902 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":298,"threads_started":1}
I20260620 18:34:03.710590 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:03.709733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:60177)
0620 18:34:03.710298 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:03.710390 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":331,"threads_started":1}
I20260620 18:34:03.822063 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:03.821167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:54519)
0620 18:34:03.821686 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:03.821806 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":223,"threads_started":1}
I20260620 18:34:04.273116 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:04.272276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:44435)
0620 18:34:04.272795 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:04.272884 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":221,"threads_started":1}
I20260620 18:34:04.712484 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:04.711622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:51395)
0620 18:34:04.712152 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:04.712242 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":245,"threads_started":1}
I20260620 18:34:04.824040 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:04.823170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:42083)
0620 18:34:04.823760 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:04.823841 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":271,"threads_started":1}
I20260620 18:34:05.274818 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:05.274100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:46219)
0620 18:34:05.274563 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:05.274655 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260620 18:34:05.714274 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:05.713499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:39949)
0620 18:34:05.714032 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:05.714108 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":329,"threads_started":1}
I20260620 18:34:05.825944 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:05.825181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:34615)
0620 18:34:05.825678 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:05.825757 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":247,"threads_started":1}
I20260620 18:34:06.276466 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:06.275746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:46361)
0620 18:34:06.276211 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:06.276289 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260620 18:34:06.715993 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:06.715198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:49119)
0620 18:34:06.715747 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:06.715827 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":241,"threads_started":1}
I20260620 18:34:06.827914 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:06.827043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:48629)
0620 18:34:06.827657 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:06.827746 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":353,"threads_started":1}
I20260620 18:34:07.278594 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:07.277594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:42973)
0620 18:34:07.278347 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:07.278426 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":400,"threads_started":1}
I20260620 18:34:07.717686 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:07.717000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:43709)
0620 18:34:07.717444 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:07.717522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260620 18:34:07.829470 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:07.828790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:46183)
0620 18:34:07.829211 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:07.829295 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":209,"threads_started":1}
I20260620 18:34:08.280315 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:08.279579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:41757)
0620 18:34:08.280027 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:08.280117 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":207,"threads_started":1}
I20260620 18:34:08.719503 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:08.718768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:35115)
0620 18:34:08.719220 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:08.719318 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260620 18:34:08.831333 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:08.830474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:46581)
0620 18:34:08.831071 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:08.831166 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260620 18:34:09.282123 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:09.281369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:51115)
0620 18:34:09.281867 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:09.281962 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260620 18:34:09.721432 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:09.720679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:39025)
0620 18:34:09.721190 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:09.721278 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":296,"threads_started":1}
I20260620 18:34:09.833503 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:09.832487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:49693)
0620 18:34:09.833132 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:09.833275 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":290,"threads_started":1}
I20260620 18:34:10.284181 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:10.283191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:33215)
0620 18:34:10.283895 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:10.283989 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":270,"threads_started":1}
I20260620 18:34:10.723438 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:10.722574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:39207)
0620 18:34:10.723123 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:10.723224 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":280,"threads_started":1}
I20260620 18:34:10.836175 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:10.834560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:34745)
0620 18:34:10.835864 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:10.835986 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":1079,"threads_started":1}
I20260620 18:34:11.286145 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:11.285336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:46937)
0620 18:34:11.285849 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:11.285946 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260620 18:34:11.725227 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:11.724453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:55551)
0620 18:34:11.724932 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:11.725019 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260620 18:34:11.838502 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:11.837424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.1:36123)
0620 18:34:11.838172 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:11.838308 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":367,"threads_started":1}
I20260620 18:34:12.288208 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:12.287319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.2:36719)
0620 18:34:12.287924 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:12.288012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":373,"threads_started":1}
I20260620 18:34:12.701484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17006
I20260620 18:34:12.724864 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:12.702656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.0.97.1:40205)
0620 18:34:12.703268 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:12.703282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:12.703319 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:34:12.705397 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:12.705407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:12.705832 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:12.706144 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:12.707731 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.708953 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.710309 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.711050 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.711198 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:12.718496 (+ 7298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:12.718505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:12.718526 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:12.718568 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:12.721596 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.722348 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.722355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.722361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.722585 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.722925 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.722928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.722930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.723447 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:12.723832 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:12.724216 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 18:34:12.724585 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":285,"threads_started":1}
I20260620 18:34:12.726332 17169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:12.727324 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:12.726331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44853 (local address 127.0.97.3:51249)
0620 18:34:12.726892 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:12.727045 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260620 18:34:12.814780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17006
I20260620 18:34:12.831589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17177
I20260620 18:34:12.850039 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:12.832562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.97.2:42637)
0620 18:34:12.833112 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:12.833126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:12.833162 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:34:12.834716 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:12.834725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:12.835163 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:12.835511 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:12.837192 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.838458 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.839821 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.840474 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.840692 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:12.843275 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:12.843283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:12.843298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:12.843336 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:12.846551 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.847302 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.847309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.847317 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.847491 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.847949 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.847954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.847958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.848519 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:12.848829 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:12.849316 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:34:12.849750 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
I20260620 18:34:12.851655 17300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:12.936316 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17177
I20260620 18:34:12.953341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17308
I20260620 18:34:12.972258 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:12.954519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.0.97.3:36129)
0620 18:34:12.955145 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:12.955160 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:12.955210 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:34:12.956933 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:12.956943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:12.957404 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:12.957703 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:12.959034 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.960589 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.961891 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:12.962587 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:12.962732 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:12.965740 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:12.965747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:12.965761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:12.965799 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:12.968900 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.969658 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.969665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.969672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.969850 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:12.970218 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:12.970222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:12.970224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:12.970795 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:12.971020 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:12.971416 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:34:12.971963 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260620 18:34:12.973686 17431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:13.075064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17308
2026-06-20T18:34:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64793 ms)
[----------] 30 tests from SecurityITest (300783 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:13 dist-test-slave-mcnm krb5kdc[17493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:13 dist-test-slave-mcnm krb5kdc[17493](info): set up 2 sockets
Jun 20 18:34:13 dist-test-slave-mcnm krb5kdc[17493](info): commencing operation
krb5kdc: starting...
W20260620 18:34:14.195685 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:14 dist-test-slave-mcnm krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:34:14.441064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:14.768118 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:14.768422 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:14.768592 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:14.776101 17506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:14.776206 17506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:14.776285 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:14.776388 17506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:14.776464 17506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:14.786427 17506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:14.787817 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:14.789201 17506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:14.801280 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.801239 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.801728 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.802302 17506 server_base.cc:1034] running on GCE node
I20260620 18:34:14.803189 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:14.804730 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:14.805987 17506 hybrid_clock.cc:648] HybridClock initialized: now 1781980454805899 us; error 36 us; skew 500 ppm
I20260620 18:34:14.806347 17506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:14 dist-test-slave-mcnm krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:14.809629 17506 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:34:14.811228 17506 webserver.cc:460] Webserver started at http://127.0.97.62:40797/ using document root <none> and password file <none>
I20260620 18:34:14.812044 17506 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:14.812194 17506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:14.812570 17506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:14.815677 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "6d0df6cd691e45f0bb9db7a1def38ad9"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-mcnm"
server_key: "a3f797f44515c7d7efd61ae1e8d41e1d"
server_key_iv: "d94066c5c809c6000f27e63f55747603"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.816455 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d0df6cd691e45f0bb9db7a1def38ad9"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-mcnm"
server_key: "a3f797f44515c7d7efd61ae1e8d41e1d"
server_key_iv: "d94066c5c809c6000f27e63f55747603"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.821637 17506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:34:14.825381 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.826754 17506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:34:14.826948 17506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "6d0df6cd691e45f0bb9db7a1def38ad9"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-mcnm"
server_key: "a3f797f44515c7d7efd61ae1e8d41e1d"
server_key_iv: "d94066c5c809c6000f27e63f55747603"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.827153 17506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:14.848510 17506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:14.852927 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:14.853250 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:14.874334 17506 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35733
I20260620 18:34:14.874353 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35733 every 8 connection(s)
I20260620 18:34:14.875870 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:14.879706 17573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:14.885704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17506
I20260620 18:34:14.885959 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:14.886677 388 external_mini_cluster.cc:1428] Setting key 89ddbdde6f3fedfdc5fc30cbc2fe3437
I20260620 18:34:14.890386 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: Bootstrap starting.
I20260620 18:34:14.893934 17573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:14.895090 17573 log.cc:824] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:14.898360 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: No bootstrap required, opened a new log
Jun 20 18:34:14 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:14.908129 17573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } }
I20260620 18:34:14.908562 17573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:14.908672 17573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0df6cd691e45f0bb9db7a1def38ad9, State: Initialized, Role: FOLLOWER
I20260620 18:34:14.909392 17573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } }
I20260620 18:34:14.909719 17573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:14.909879 17573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:14.910068 17573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:14.912660 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.889047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.0.97.62:35733)
0620 18:34:14.889855 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.889867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.889908 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:34:14.891795 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.891822 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.892205 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.892538 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.894360 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.895914 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.898042 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.899081 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.899355 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.904474 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.904483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.904499 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.904548 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.908020 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.908795 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.908810 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.908817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.909014 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.909468 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.909471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.909473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.910213 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.910461 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.911013 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.911461 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260620 18:34:14.913620 17573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } }
I20260620 18:34:14.914223 17573 leader_election.cc:302] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d0df6cd691e45f0bb9db7a1def38ad9; no voters:
I20260620 18:34:14.914853 17573 leader_election.cc:288] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:14.915077 17578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:14.916616 17578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d0df6cd691e45f0bb9db7a1def38ad9, State: Running, Role: LEADER
I20260620 18:34:14.917135 17573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:14.917114 17578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } }
I20260620 18:34:14.920095 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0df6cd691e45f0bb9db7a1def38ad9. Latest consensus state: current_term: 1 leader_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } } }
I20260620 18:34:14.920445 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:14.921444 17579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0df6cd691e45f0bb9db7a1def38ad9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35733 } } }
I20260620 18:34:14.922461 17579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:14.928277 17586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:14.934158 17586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:14.943203 17586 catalog_manager.cc:1347] Generated new cluster ID: 5970b6dfcd89442ba1dbf503f2399c60
I20260620 18:34:14.943346 17586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:14.974931 17586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:14.976326 17586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:14.986057 17586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: Generated new TSK 0
I20260620 18:34:14.986934 17586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:34:15.047360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:15.378328 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:15.378630 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:15.378728 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:15.386070 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:15.386191 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:15.386389 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:15.396346 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:15.397732 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:15.399084 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:15.412693 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.412616 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.412669 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:15.413882 17599 server_base.cc:1034] running on GCE node
I20260620 18:34:15.414518 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:15.415479 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:15.416769 17599 hybrid_clock.cc:648] HybridClock initialized: now 1781980455416711 us; error 41 us; skew 500 ppm
I20260620 18:34:15.417239 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:15 dist-test-slave-mcnm krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:15.420845 17599 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:34:15.422421 17599 webserver.cc:460] Webserver started at http://127.0.97.1:39053/ using document root <none> and password file <none>
I20260620 18:34:15.423130 17599 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:15.423274 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:15.423650 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:15.426553 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "d4a92597cdf043a2b3eded1c3fbc9b95"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "301c66c4ed53fe90b63e31648fcbbb6a"
server_key_iv: "a24b2dca77037174ba82676903c9f5f1"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.427418 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d4a92597cdf043a2b3eded1c3fbc9b95"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "301c66c4ed53fe90b63e31648fcbbb6a"
server_key_iv: "a24b2dca77037174ba82676903c9f5f1"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.432744 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:15.436204 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.437590 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:15.437865 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "d4a92597cdf043a2b3eded1c3fbc9b95"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "301c66c4ed53fe90b63e31648fcbbb6a"
server_key_iv: "a24b2dca77037174ba82676903c9f5f1"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.438071 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:15.453966 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:15.458968 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:15.459301 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:15.460412 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:15.461826 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:15.461911 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.462107 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:15.462167 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.498155 17599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41505
I20260620 18:34:15.498193 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41505 every 8 connection(s)
I20260620 18:34:15.499490 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:15.501334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17599
I20260620 18:34:15.501554 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:15.502363 388 external_mini_cluster.cc:1428] Setting key 1a364ceec779d4ba9c141b4ea5e19140
Jun 20 18:34:15 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:15.519866 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:15.502282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50401 (local address 127.0.97.62:35733)
0620 18:34:15.502830 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:15.502836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:15.503820 (+ 984us) server_negotiation.cc:407] Connection header received
0620 18:34:15.505660 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:15.505665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:15.505752 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:15.505868 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:15.507393 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.508163 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.509713 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.510108 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.510241 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:15.513790 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:15.513797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:15.513802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:15.513871 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:15.516702 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.517498 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.517504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.517507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.517634 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.518070 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.518076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.518080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.518361 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:15.518523 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:15.519237 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 18:34:15.519476 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260620 18:34:15.520813 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:15.502833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35733 (local address 127.0.97.1:50401)
0620 18:34:15.503651 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:15.503697 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:15.505395 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:15.506133 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:15.506142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:15.506630 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:15.507212 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:15.507229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.508342 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.508346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:15.509555 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:15.509566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.510345 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.510353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:15.510590 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:15.511282 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:15.511304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:15.513547 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:15.516899 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:15.516908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:15.516924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:15.517361 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:15.517751 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:15.517756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:15.517760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:15.517956 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:15.518494 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:15.518501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:15.518870 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:34:15.519140 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 18:34:15.519682 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":332,"threads_started":1}
I20260620 18:34:15.522397 17727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35733
I20260620 18:34:15.522704 17727 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:15.523433 17727 heartbeater.cc:511] Master 127.0.97.62:35733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 18:34:15.525624 17538 ts_manager.cc:194] Registered new tserver with Master: d4a92597cdf043a2b3eded1c3fbc9b95 (127.0.97.1:41505)
I20260620 18:34:15.527284 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:34:15.559983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:15.884912 17733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:15.885238 17733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:15.885437 17733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:15.892948 17733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:15.893118 17733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:15.893312 17733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:15.903375 17733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:15.904765 17733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:15.906193 17733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:15.919584 17738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.919659 17739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.920282 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:15.925330 17733 server_base.cc:1034] running on GCE node
I20260620 18:34:15.926009 17733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:15.926988 17733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:15.928200 17733 hybrid_clock.cc:648] HybridClock initialized: now 1781980455928153 us; error 92 us; skew 500 ppm
I20260620 18:34:15.928608 17733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:15 dist-test-slave-mcnm krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:15.932277 17733 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:34:15.933980 17733 webserver.cc:460] Webserver started at http://127.0.97.2:46083/ using document root <none> and password file <none>
I20260620 18:34:15.934725 17733 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:15.934870 17733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:15.935220 17733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:15.938251 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "5dcb6c5e31bbca8182a539ec26b04f41"
server_key_iv: "e6ba7ce46d642e5ff669694f43ab43de"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.939047 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "5dcb6c5e31bbca8182a539ec26b04f41"
server_key_iv: "e6ba7ce46d642e5ff669694f43ab43de"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.944567 17733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 18:34:15.948449 17748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.949955 17733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:15.950100 17733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-mcnm"
server_key: "5dcb6c5e31bbca8182a539ec26b04f41"
server_key_iv: "e6ba7ce46d642e5ff669694f43ab43de"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.950273 17733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:15.980139 17733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:15.984124 17733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:15.984361 17733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:15.985265 17733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:15.986545 17733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:15.986617 17733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.986697 17733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:15.986748 17733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.021333 17733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39567
I20260620 18:34:16.021378 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39567 every 8 connection(s)
I20260620 18:34:16.022639 17733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:16.023773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17733
I20260620 18:34:16.024010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:16.024546 388 external_mini_cluster.cc:1428] Setting key 77e146741b91e0aba88f13c60c9a656b
Jun 20 18:34:16 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:16.048162 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.026147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55127 (local address 127.0.97.62:35733)
0620 18:34:16.028351 (+ 2204us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.028360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.028901 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:34:16.031452 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.031458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.031619 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.031746 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.033381 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.034128 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.036277 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.036774 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.036884 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.041443 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.041448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.041452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.041490 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.044571 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.045428 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.045433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.045436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.045581 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.046107 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.046112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.046114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.046522 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.046693 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.047467 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.047745 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":1948,"threads_started":1}
I20260620 18:34:16.049228 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.026609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35733 (local address 127.0.97.2:55127)
0620 18:34:16.028734 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.028783 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.031138 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.032018 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.032051 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.032620 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:16.033207 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.033225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.034300 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.034304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.036082 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.036098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.037022 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.037032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.037375 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.038154 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:16.038189 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:16.041162 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:16.044747 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.044758 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.044778 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.045264 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.045705 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.045712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.045716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.045941 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.046661 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:16.046671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:16.047069 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.047355 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.047985 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1874,"thread_start_us":1681,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 18:34:16.050843 17861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35733
I20260620 18:34:16.051213 17861 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:16.051964 17861 heartbeater.cc:511] Master 127.0.97.62:35733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 18:34:16.053719 17538 ts_manager.cc:194] Registered new tserver with Master: c5dcd00cedda4cc7a6207c8a952c7b7f (127.0.97.2:39567)
I20260620 18:34:16.055099 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:34:16.083501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:16.415124 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:16.415442 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:16.415654 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:16.423180 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:16.423295 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:16.423410 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:16.433454 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:16.434719 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:16.435992 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:16.449328 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.449337 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.449635 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.450047 17867 server_base.cc:1034] running on GCE node
I20260620 18:34:16.450703 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:16.451682 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:16.452909 17867 hybrid_clock.cc:648] HybridClock initialized: now 1781980456452862 us; error 57 us; skew 500 ppm
I20260620 18:34:16.453234 17867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:16 dist-test-slave-mcnm krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:16.456589 17867 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:34:16.458067 17867 webserver.cc:460] Webserver started at http://127.0.97.3:45863/ using document root <none> and password file <none>
I20260620 18:34:16.458779 17867 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:16.458945 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:16.459280 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:16.462539 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "808d8d9b638d410c83ef95fee0dc791d"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-mcnm"
server_key: "a74dcf7c4c10d7fbfc90cb23772418b1"
server_key_iv: "13a1d34ac585578e8b32c2a908d44b8f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.463387 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "808d8d9b638d410c83ef95fee0dc791d"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-mcnm"
server_key: "a74dcf7c4c10d7fbfc90cb23772418b1"
server_key_iv: "13a1d34ac585578e8b32c2a908d44b8f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.468957 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:34:16.472918 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.474274 17867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:16.474464 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "808d8d9b638d410c83ef95fee0dc791d"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-mcnm"
server_key: "a74dcf7c4c10d7fbfc90cb23772418b1"
server_key_iv: "13a1d34ac585578e8b32c2a908d44b8f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.474699 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:16.491009 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:16.495004 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:16.495311 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:16.496392 17867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:16.497706 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:16.497807 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.497957 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:16.498029 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.530505 17727 heartbeater.cc:503] Master 127.0.97.62:35733 was elected leader, sending a full tablet report...
I20260620 18:34:16.532354 17867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43835
I20260620 18:34:16.532445 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43835 every 8 connection(s)
I20260620 18:34:16.533752 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:16.537169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 17867
I20260620 18:34:16.537356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:16.537837 388 external_mini_cluster.cc:1428] Setting key 8d67e556663afdd1d6bae1095d0e329b
Jun 20 18:34:16 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:16.553891 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.536217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36141 (local address 127.0.97.62:35733)
0620 18:34:16.536459 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.536464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.537807 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 18:34:16.539819 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.539831 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.539910 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.539988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.541636 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.542447 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.544027 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.544435 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.544575 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.547918 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.547923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.547926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.547960 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.550630 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.551395 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.551400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.551402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.551625 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.552139 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.552143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.552145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.552512 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.552645 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.553482 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.553662 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 18:34:16.555053 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.536799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35733 (local address 127.0.97.3:36141)
0620 18:34:16.537634 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.537680 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.539556 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.540269 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.540280 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.540880 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:16.541489 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.541508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.542610 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.542615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.543888 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.543900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.544659 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.544668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.544895 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.545531 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:16.545557 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:16.547714 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:16.550767 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.550777 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.550794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.551239 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.551745 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.551750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.551754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.551992 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.552656 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:16.552666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:16.553067 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.553380 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.553801 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260620 18:34:16.556504 17995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35733
I20260620 18:34:16.556767 17995 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:16.557432 17995 heartbeater.cc:511] Master 127.0.97.62:35733 requested a full tablet report, sending...
I20260620 18:34:16.558756 17538 ts_manager.cc:194] Registered new tserver with Master: 808d8d9b638d410c83ef95fee0dc791d (127.0.97.3:43835)
I20260620 18:34:16.559618 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36141
I20260620 18:34:16.565332 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:16.583098 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.569948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.0.97.62:35733)
0620 18:34:16.570123 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.570127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.570572 (+ 445us) server_negotiation.cc:407] Connection header received
0620 18:34:16.570714 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.570718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.570779 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.570873 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.572310 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.573040 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.574431 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.574814 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.574908 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.577320 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.577324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.577329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.577405 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.580419 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.581171 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.581175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.581178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.581323 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.581679 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.581683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.581685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.582038 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.582239 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.582620 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.582791 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:34:16.587692 17538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:16.595108 17538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:16.631696 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.609062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.97.3:43835)
0620 18:34:16.621882 (+ 12820us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.621891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.621930 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:34:16.622022 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.622032 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.622270 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.622476 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.623589 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.625123 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.628623 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.630005 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.630114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.630350 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.631106 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.631284 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12618,"thread_start_us":12467,"threads_started":1}
I20260620 18:34:16.631705 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.608321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.1:41505)
0620 18:34:16.620252 (+ 11931us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.620263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.621216 (+ 953us) server_negotiation.cc:407] Connection header received
0620 18:34:16.621335 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.621343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.621648 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.622572 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.622806 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.624535 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.627792 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.629991 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.630114 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.630348 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.631194 (+ 846us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.631353 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11726,"thread_start_us":11540,"threads_started":1}
I20260620 18:34:16.631815 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.608029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52614)
0620 18:34:16.620715 (+ 12686us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.620774 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.620995 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.621917 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.621925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.621961 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.622369 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.622387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.624701 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.624706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.627613 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.627641 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.630609 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.630614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.630811 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.630905 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.631044 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.631269 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12372,"mutex_wait_us":190,"spinlock_wait_cycles":3712,"thread_start_us":13126,"threads_started":1}
I20260620 18:34:16.632527 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.622146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.0.97.2:39567)
0620 18:34:16.622784 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.622793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.623789 (+ 996us) server_negotiation.cc:407] Connection header received
0620 18:34:16.624175 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.624190 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.624347 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.624547 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.625194 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.626430 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.629209 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.630440 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.630594 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.630814 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.631551 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.631719 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":367,"threads_started":1}
I20260620 18:34:16.633183 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.622865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39567 (local address 127.0.0.1:53422)
0620 18:34:16.623654 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.623679 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.623905 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.624534 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.624539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.624583 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.625004 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.625014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.626608 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.626613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.629043 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.629058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.631114 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.631122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.631316 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.631349 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.632725 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.632829 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":59,"thread_start_us":448,"threads_started":1}
I20260620 18:34:16.633364 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.608431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43835 (local address 127.0.0.1:38936)
0620 18:34:16.620715 (+ 12284us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.620774 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.620995 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.622751 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.622756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.622783 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.623336 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.623348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.625263 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.625269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.627651 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.627661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.630528 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.630537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.630806 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.630916 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.631044 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.632879 (+ 1835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12124,"mutex_wait_us":908,"spinlock_wait_cycles":5376,"thread_start_us":12020,"threads_started":1}
I20260620 18:34:16.635828 17662 tablet_service.cc:1467] Processing CreateTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 (DEFAULT_TABLE table=test-table [id=3def1dc8e9104d159a4ae2590d243275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.636691 17796 tablet_service.cc:1467] Processing CreateTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 (DEFAULT_TABLE table=test-table [id=3def1dc8e9104d159a4ae2590d243275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.636902 17930 tablet_service.cc:1467] Processing CreateTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 (DEFAULT_TABLE table=test-table [id=3def1dc8e9104d159a4ae2590d243275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.637351 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c1bf35c92a4826994dd2a448816ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.638002 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c1bf35c92a4826994dd2a448816ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.638083 17796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c1bf35c92a4826994dd2a448816ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.649580 18013 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Bootstrap starting.
I20260620 18:34:16.653141 18013 tablet_bootstrap.cc:654] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.653993 18014 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Bootstrap starting.
I20260620 18:34:16.654666 18013 log.cc:824] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.655102 18015 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Bootstrap starting.
I20260620 18:34:16.657673 18013 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: No bootstrap required, opened a new log
I20260620 18:34:16.657989 18013 ts_tablet_manager.cc:1397] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 18:34:16.658877 18015 tablet_bootstrap.cc:654] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.658880 18014 tablet_bootstrap.cc:654] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.660163 18015 log.cc:824] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.660162 18014 log.cc:824] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.662909 18014 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: No bootstrap required, opened a new log
I20260620 18:34:16.663218 18014 ts_tablet_manager.cc:1397] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:34:16.663420 18015 tablet_bootstrap.cc:492] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: No bootstrap required, opened a new log
I20260620 18:34:16.663729 18015 ts_tablet_manager.cc:1397] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 18:34:16.668360 18013 raft_consensus.cc:348] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.668833 18013 raft_consensus.cc:374] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.668951 18013 raft_consensus.cc:729] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808d8d9b638d410c83ef95fee0dc791d, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.669632 18013 consensus_queue.cc:260] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.670785 17995 heartbeater.cc:503] Master 127.0.97.62:35733 was elected leader, sending a full tablet report...
I20260620 18:34:16.671455 18013 ts_tablet_manager.cc:1428] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260620 18:34:16.671815 18014 raft_consensus.cc:348] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.672240 18014 raft_consensus.cc:374] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.672363 18014 raft_consensus.cc:729] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a92597cdf043a2b3eded1c3fbc9b95, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.672933 18014 consensus_queue.cc:260] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.673995 18015 raft_consensus.cc:348] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.674401 18015 raft_consensus.cc:374] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.674502 18015 raft_consensus.cc:729] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dcd00cedda4cc7a6207c8a952c7b7f, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.674639 18014 ts_tablet_manager.cc:1428] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260620 18:34:16.675067 18015 consensus_queue.cc:260] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.676209 17861 heartbeater.cc:503] Master 127.0.97.62:35733 was elected leader, sending a full tablet report...
I20260620 18:34:16.676898 18015 ts_tablet_manager.cc:1428] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260620 18:34:16.752135 17728 tablet.cc:2369] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:16.775178 17862 tablet.cc:2369] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:16.785353 17996 tablet.cc:2369] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:16.838521 18020 raft_consensus.cc:482] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:16.838855 18020 raft_consensus.cc:504] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.840535 18020 leader_election.cc:288] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 808d8d9b638d410c83ef95fee0dc791d (127.0.97.3:43835), c5dcd00cedda4cc7a6207c8a952c7b7f (127.0.97.2:39567)
I20260620 18:34:16.847620 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.841074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59535 (local address 127.0.97.3:43835)
0620 18:34:16.841277 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.841281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.841851 (+ 570us) server_negotiation.cc:407] Connection header received
0620 18:34:16.842094 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.842103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.842208 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.842294 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.842990 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.844092 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.845883 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.846674 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.846828 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.846935 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.847218 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.847344 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:34:16.847730 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.841003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43835 (local address 127.0.97.1:59535)
0620 18:34:16.841726 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.841748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.841891 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.842517 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.842521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.842542 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.842857 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.842864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.844243 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.844247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.845716 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.845729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.846884 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.846890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.847045 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.847070 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.847178 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.847322 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":401,"threads_started":1}
I20260620 18:34:16.848201 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.842166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57733 (local address 127.0.97.2:39567)
0620 18:34:16.842416 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.842423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.842649 (+ 226us) server_negotiation.cc:407] Connection header received
0620 18:34:16.842878 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.842884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.842984 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.843080 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.843591 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.844498 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.846347 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.847198 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.847364 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.847469 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.847860 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.847987 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:34:16.848871 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.841848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39567 (local address 127.0.97.1:57733)
0620 18:34:16.842525 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.842543 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.842736 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.843102 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.843105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.843123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.843383 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.843389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.844645 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.844650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.846061 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.846182 (+ 121us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.847431 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.847437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.847648 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.847670 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.848353 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.848494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"spinlock_wait_cycles":1408,"thread_start_us":345,"threads_started":1}
I20260620 18:34:16.848816 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c1bf35c92a4826994dd2a448816ba2" candidate_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808d8d9b638d410c83ef95fee0dc791d" is_pre_election: true
I20260620 18:34:16.849381 17950 raft_consensus.cc:2454] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a92597cdf043a2b3eded1c3fbc9b95 in term 0.
I20260620 18:34:16.849952 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c1bf35c92a4826994dd2a448816ba2" candidate_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" is_pre_election: true
I20260620 18:34:16.850325 17618 leader_election.cc:302] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808d8d9b638d410c83ef95fee0dc791d, d4a92597cdf043a2b3eded1c3fbc9b95; no voters:
I20260620 18:34:16.850554 17816 raft_consensus.cc:2454] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a92597cdf043a2b3eded1c3fbc9b95 in term 0.
I20260620 18:34:16.850762 18020 raft_consensus.cc:2781] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:16.850911 18020 raft_consensus.cc:482] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:16.851039 18020 raft_consensus.cc:3037] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.853547 18020 raft_consensus.cc:504] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.854159 18020 leader_election.cc:288] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [CANDIDATE]: Term 1 election: Requested vote from peers 808d8d9b638d410c83ef95fee0dc791d (127.0.97.3:43835), c5dcd00cedda4cc7a6207c8a952c7b7f (127.0.97.2:39567)
I20260620 18:34:16.854765 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c1bf35c92a4826994dd2a448816ba2" candidate_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808d8d9b638d410c83ef95fee0dc791d"
I20260620 18:34:16.854797 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c1bf35c92a4826994dd2a448816ba2" candidate_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f"
I20260620 18:34:16.854993 17950 raft_consensus.cc:3037] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.855000 17816 raft_consensus.cc:3037] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.858242 17816 raft_consensus.cc:2454] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a92597cdf043a2b3eded1c3fbc9b95 in term 1.
I20260620 18:34:16.858242 17950 raft_consensus.cc:2454] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a92597cdf043a2b3eded1c3fbc9b95 in term 1.
I20260620 18:34:16.858646 17618 leader_election.cc:302] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5dcd00cedda4cc7a6207c8a952c7b7f, d4a92597cdf043a2b3eded1c3fbc9b95; no voters:
I20260620 18:34:16.858999 18020 raft_consensus.cc:2781] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:16.859459 18020 raft_consensus.cc:686] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 1 LEADER]: Becoming Leader. State: Replica: d4a92597cdf043a2b3eded1c3fbc9b95, State: Running, Role: LEADER
I20260620 18:34:16.860073 18020 consensus_queue.cc:237] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } }
I20260620 18:34:16.864645 17536 catalog_manager.cc:5495] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4a92597cdf043a2b3eded1c3fbc9b95 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4a92597cdf043a2b3eded1c3fbc9b95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:16.935863 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.930290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.0.97.1:41505)
0620 18:34:16.930472 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.930476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.930493 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:34:16.930649 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.930654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.930788 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.930917 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:16.931267 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.932165 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.933593 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.934011 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.934106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.934576 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:16.935204 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:16.935314 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.935585 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.935660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:34:16.944473 17950 raft_consensus.cc:1261] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 1 FOLLOWER]: Refusing update from remote peer d4a92597cdf043a2b3eded1c3fbc9b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:16.944485 17816 raft_consensus.cc:1261] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 1 FOLLOWER]: Refusing update from remote peer d4a92597cdf043a2b3eded1c3fbc9b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:16.945498 18024 consensus_queue.cc:1035] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dcd00cedda4cc7a6207c8a952c7b7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:16.945945 18020 consensus_queue.cc:1035] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808d8d9b638d410c83ef95fee0dc791d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:16.957500 18027 mvcc.cc:205] Tried to move back new op lower bound from 7298991951632846848 to 7298991951302696960. Current Snapshot: MvccSnapshot[applied={T|T < 7298991951632846848}]
I20260620 18:34:16.959652 18029 mvcc.cc:205] Tried to move back new op lower bound from 7298991951632846848 to 7298991951302696960. Current Snapshot: MvccSnapshot[applied={T|T < 7298991951632846848}]
I20260620 18:34:16.974045 17536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:16.974331 17536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:16.977872 17536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d0df6cd691e45f0bb9db7a1def38ad9: Sending DeleteTablet for 3 replicas of tablet 88c1bf35c92a4826994dd2a448816ba2
I20260620 18:34:16.979130 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52614
I20260620 18:34:16.979212 17930 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38936
I20260620 18:34:16.979246 17796 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c1bf35c92a4826994dd2a448816ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53422
I20260620 18:34:16.981815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17599
I20260620 18:34:16.992046 18037 tablet_replica.cc:330] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: stopping tablet replica
I20260620 18:34:16.992352 18038 tablet_replica.cc:330] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: stopping tablet replica
I20260620 18:34:16.992849 18038 raft_consensus.cc:2229] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:16.993371 18038 raft_consensus.cc:2258] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:16.995832 18038 ts_tablet_manager.cc:1905] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:16.997777 18039 tablet_replica.cc:330] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: stopping tablet replica
I20260620 18:34:16.998319 18039 raft_consensus.cc:2229] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:16.999181 18039 raft_consensus.cc:2258] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 18:34:16 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:34:17.002139 18039 ts_tablet_manager.cc:1905] T 88c1bf35c92a4826994dd2a448816ba2 P d4a92597cdf043a2b3eded1c3fbc9b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:17.002724 18037 raft_consensus.cc:2229] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:17.003897 18038 ts_tablet_manager.cc:1918] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:17.004074 18038 log.cc:1196] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/88c1bf35c92a4826994dd2a448816ba2
I20260620 18:34:17.004654 18037 raft_consensus.cc:2258] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:17.004923 18038 ts_tablet_manager.cc:1939] T 88c1bf35c92a4826994dd2a448816ba2 P c5dcd00cedda4cc7a6207c8a952c7b7f: Deleting consensus metadata
I20260620 18:34:17.006924 17525 catalog_manager.cc:4841] TS c5dcd00cedda4cc7a6207c8a952c7b7f (127.0.97.2:39567): tablet 88c1bf35c92a4826994dd2a448816ba2 (table test-table [id=3def1dc8e9104d159a4ae2590d243275]) successfully deleted
I20260620 18:34:17.010823 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.983413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.97.1:41505)
0620 18:34:16.983665 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.983675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.988106 (+ 4431us) server_negotiation.cc:407] Connection header received
0620 18:34:16.988902 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.988911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.989021 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.989119 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.992273 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.993529 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.995675 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.996369 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.996471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.002304 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.002319 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.002337 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.002407 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.006998 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.008238 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.008248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.008251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.008466 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.008979 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.008987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.008992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.009492 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:17.009839 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:17.010166 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:34:17.010428 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:34:17.011696 18037 ts_tablet_manager.cc:1905] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:17.011714 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:17.019980 18037 ts_tablet_manager.cc:1918] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:17.020169 18037 log.cc:1196] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/88c1bf35c92a4826994dd2a448816ba2
I20260620 18:34:17.020696 18037 ts_tablet_manager.cc:1939] T 88c1bf35c92a4826994dd2a448816ba2 P 808d8d9b638d410c83ef95fee0dc791d: Deleting consensus metadata
I20260620 18:34:17.022724 17525 catalog_manager.cc:4841] TS 808d8d9b638d410c83ef95fee0dc791d (127.0.97.3:43835): tablet 88c1bf35c92a4826994dd2a448816ba2 (table test-table [id=3def1dc8e9104d159a4ae2590d243275]) successfully deleted
I20260620 18:34:17.112561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17599
I20260620 18:34:17.129377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17733
W20260620 18:34:17.129442 17522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41505 (error 108)
I20260620 18:34:17.131835 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.131235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52642)
0620 18:34:17.131453 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.131626 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 18:34:17.132088 17522 catalog_manager.cc:4584] TS d4a92597cdf043a2b3eded1c3fbc9b95 (127.0.97.1:41505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88c1bf35c92a4826994dd2a448816ba2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Jun 20 18:34:17 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:34:17.145480 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.130239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.97.2:39567)
0620 18:34:17.130474 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:17.130480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:17.130502 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:34:17.130705 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:17.130710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:17.130815 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:17.130899 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:17.132771 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.133604 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.135083 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.135469 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.135615 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.139821 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.139835 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.139849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.139883 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.142533 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.143211 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.143219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.143221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.143357 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.144015 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.144019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.144021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.144434 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:17.144738 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:17.145023 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:34:17.145197 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:34:17.146240 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:17.185770 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.185276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52656)
0620 18:34:17.185499 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.185590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:34:17.186899 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.186554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52664)
0620 18:34:17.186714 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.186793 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:34:17.242208 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17733
I20260620 18:34:17.247669 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.247048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52670)
0620 18:34:17.247307 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.247433 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:34:17.248816 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.248370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52684)
0620 18:34:17.248544 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.248652 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:34:17.260079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17867
Jun 20 18:34:17 dist-test-slave-mcnm krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:34:17.274384 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.260844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.0.97.3:43835)
0620 18:34:17.261026 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:17.261030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:17.261048 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:34:17.261202 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:17.261206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:17.261285 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:17.261371 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:17.262756 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.263465 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.264796 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.265183 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.265278 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.269448 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.269463 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.269478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.269511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.272099 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.272697 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.272704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.272706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.272845 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.273140 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.273154 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.273156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.273462 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:17.273686 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:17.273964 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:34:17.274119 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:17.275027 17990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:17.330135 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.329651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52690)
0620 18:34:17.329865 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.329956 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:34:17.331394 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.331000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:52702)
0620 18:34:17.331196 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.331283 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 18:34:17.364419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17867
I20260620 18:34:17.382803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 17506
I20260620 18:34:17.384074 17568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:17.494511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 17506
2026-06-20T18:34:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4422 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:17Z Disabled control of system clock
I20260620 18:34:17.572573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46215
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:17.896356 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:17.896647 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:17.896806 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:17.904038 18049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:17.904141 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:17.904227 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:17.904362 18049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:17.904448 18049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:17.913771 18049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:17.915114 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:17.916958 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:17.929768 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:17.929844 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:17.929821 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:17.930485 18049 server_base.cc:1034] running on GCE node
I20260620 18:34:17.931285 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:17.932806 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:17.934062 18049 hybrid_clock.cc:648] HybridClock initialized: now 1781980457934006 us; error 50 us; skew 500 ppm
I20260620 18:34:17.934509 18049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:17.942764 18049 webserver.cc:460] Webserver started at http://127.0.97.62:36061/ using document root <none> and password file <none>
I20260620 18:34:17.943591 18049 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:17.943730 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:17.944063 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:17.947016 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "4e4d5761dd5a4397a4dc143c9600b9b7"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-mcnm"
server_key: "1f8aabb1a3f79061b5972aa07ba1e005"
server_key_iv: "e652e58b28ada5e2bc7e9f0700cc829b"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.947937 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e4d5761dd5a4397a4dc143c9600b9b7"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-mcnm"
server_key: "1f8aabb1a3f79061b5972aa07ba1e005"
server_key_iv: "e652e58b28ada5e2bc7e9f0700cc829b"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.953133 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:17.956508 18063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.957862 18049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:34:17.958050 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "4e4d5761dd5a4397a4dc143c9600b9b7"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-mcnm"
server_key: "1f8aabb1a3f79061b5972aa07ba1e005"
server_key_iv: "e652e58b28ada5e2bc7e9f0700cc829b"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.958261 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:17.975804 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:17.976723 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:17.977037 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:17.996452 18049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46215
I20260620 18:34:17.996464 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46215 every 8 connection(s)
I20260620 18:34:17.998178 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:18.002169 18115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.004840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18049
I20260620 18:34:18.005020 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:18.005597 388 external_mini_cluster.cc:1428] Setting key 35a0819b89ddba4b9fbd008a518bca2f
I20260620 18:34:18.014808 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: Bootstrap starting.
I20260620 18:34:18.018457 18115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.019726 18115 log.cc:824] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.020228 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.007983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.97.62:46215)
0620 18:34:18.009062 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.009075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.009119 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:34:18.010926 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.010949 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.011456 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.011886 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.012487 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.013768 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.015658 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.016376 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.016518 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.017788 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.017796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.017809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:18.018349 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.018458 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:18.018480 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.018618 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.018851 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.019148 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":406,"threads_started":1}
I20260620 18:34:18.022611 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: No bootstrap required, opened a new log
I20260620 18:34:18.029732 18115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } }
I20260620 18:34:18.030115 18115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.030229 18115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4d5761dd5a4397a4dc143c9600b9b7, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.030782 18115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } }
I20260620 18:34:18.031030 18115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:18.031178 18115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:18.031366 18115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.033578 18115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } }
I20260620 18:34:18.034142 18115 leader_election.cc:302] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e4d5761dd5a4397a4dc143c9600b9b7; no voters:
I20260620 18:34:18.034825 18115 leader_election.cc:288] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:18.035035 18120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:18.036347 18120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4d5761dd5a4397a4dc143c9600b9b7, State: Running, Role: LEADER
I20260620 18:34:18.036906 18120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } }
I20260620 18:34:18.037119 18115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:18.040372 18122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4d5761dd5a4397a4dc143c9600b9b7. Latest consensus state: current_term: 1 leader_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } } }
I20260620 18:34:18.040869 18122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:18.040923 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d5761dd5a4397a4dc143c9600b9b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46215 } } }
I20260620 18:34:18.041255 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:18.043081 18129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:18.048034 18129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:18.056846 18129 catalog_manager.cc:1347] Generated new cluster ID: 6cf4b83a0a334d558693f01bc1641669
I20260620 18:34:18.056995 18129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:18.079128 18129 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:18.080482 18129 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:18.088671 18129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: Generated new TSK 0
I20260620 18:34:18.089725 18129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:18.106652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--builtin_ntp_servers=127.0.97.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:18.441434 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.441764 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.441928 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.449361 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.449512 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.449717 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:18.459036 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:18.460378 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.462285 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.475755 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.475844 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.475723 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.476487 18139 server_base.cc:1034] running on GCE node
I20260620 18:34:18.477128 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.478160 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.479393 18139 hybrid_clock.cc:648] HybridClock initialized: now 1781980458479342 us; error 39 us; skew 500 ppm
I20260620 18:34:18.479950 18139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:18.482236 18139 webserver.cc:460] Webserver started at http://127.0.97.1:39721/ using document root <none> and password file <none>
I20260620 18:34:18.482932 18139 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.483048 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.483366 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:18.486153 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "2bf2f4c74d7b4d07bc04367153ddd033"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "3c03adc140a0133a4375d7ed33cf23eb"
server_key_iv: "5ae698e6a282665425be1ff2add4f665"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.486941 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf2f4c74d7b4d07bc04367153ddd033"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "3c03adc140a0133a4375d7ed33cf23eb"
server_key_iv: "5ae698e6a282665425be1ff2add4f665"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.491879 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:34:18.495167 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.496528 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:18.496722 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "2bf2f4c74d7b4d07bc04367153ddd033"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "3c03adc140a0133a4375d7ed33cf23eb"
server_key_iv: "5ae698e6a282665425be1ff2add4f665"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.496946 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.528527 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.529397 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.529683 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.530658 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:18.532248 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.532364 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.532493 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.532583 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.565696 18139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35651
I20260620 18:34:18.565774 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35651 every 8 connection(s)
I20260620 18:34:18.567027 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:18.570333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18139
I20260620 18:34:18.570538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:18.571110 388 external_mini_cluster.cc:1428] Setting key 162987eb6a8a3910695ffdc719e509c1
I20260620 18:34:18.574100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--builtin_ntp_servers=127.0.97.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:18.581126 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.569607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43243 (local address 127.0.97.62:46215)
0620 18:34:18.570093 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.570101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.571396 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 18:34:18.573298 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.573303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.573365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.573455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.574930 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.575787 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.578209 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.578629 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.578736 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.580354 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.580359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.580362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:18.580381 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.580421 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:18.580433 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.580507 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.580775 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.580857 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
I20260620 18:34:18.582527 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.570127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46215 (local address 127.0.97.1:43243)
0620 18:34:18.571231 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.571278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.573008 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.573742 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.573754 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.573881 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.574711 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.574738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.575946 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.575954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.578048 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.578065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.578869 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.578878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.579224 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.580011 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:18.580017 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.580071 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:18.580168 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.580540 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.580553 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.580563 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.580751 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.581166 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":312,"threads_started":1}
I20260620 18:34:18.584052 18266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46215
I20260620 18:34:18.584353 18266 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.585057 18266 heartbeater.cc:511] Master 127.0.97.62:46215 requested a full tablet report, sending...
I20260620 18:34:18.587057 18080 ts_manager.cc:194] Registered new tserver with Master: 2bf2f4c74d7b4d07bc04367153ddd033 (127.0.97.1:35651)
I20260620 18:34:18.588794 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43243
W20260620 18:34:18.907756 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.908046 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.908135 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.915875 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.915992 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.916115 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:18.925232 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:18.926540 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.928601 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.942121 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.942449 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.942567 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.943820 18270 server_base.cc:1034] running on GCE node
I20260620 18:34:18.944516 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.945524 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.946861 18270 hybrid_clock.cc:648] HybridClock initialized: now 1781980458946812 us; error 63 us; skew 500 ppm
I20260620 18:34:18.947252 18270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:18.949744 18270 webserver.cc:460] Webserver started at http://127.0.97.2:42221/ using document root <none> and password file <none>
I20260620 18:34:18.950466 18270 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.950603 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.950940 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:18.953878 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "2fe2edf5a6d54159bce7f1376938b756"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "bba58ab3ff5f47d1b6f7cd2caf9e0ab2"
server_key_iv: "56c9ff97b31260977eaa24150f369583"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.954687 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2fe2edf5a6d54159bce7f1376938b756"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "bba58ab3ff5f47d1b6f7cd2caf9e0ab2"
server_key_iv: "56c9ff97b31260977eaa24150f369583"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.960018 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:34:18.963670 18284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.964980 18270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:18.965159 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "2fe2edf5a6d54159bce7f1376938b756"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-mcnm"
server_key: "bba58ab3ff5f47d1b6f7cd2caf9e0ab2"
server_key_iv: "56c9ff97b31260977eaa24150f369583"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.965391 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.992725 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.993572 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.993855 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.994892 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:18.996465 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.996591 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.996732 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.996822 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:19.031263 18270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44223
I20260620 18:34:19.031355 18396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44223 every 8 connection(s)
I20260620 18:34:19.032570 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:19.040889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18270
I20260620 18:34:19.041123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:19.041663 388 external_mini_cluster.cc:1428] Setting key 918fa099d5756dfb9cdde70685b42098
I20260620 18:34:19.044714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--builtin_ntp_servers=127.0.97.20:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:19.045653 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.035169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47461 (local address 127.0.97.62:46215)
0620 18:34:19.035362 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.035367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.036880 (+ 1513us) server_negotiation.cc:407] Connection header received
0620 18:34:19.038881 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.038885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.038953 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.039033 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.040126 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.040832 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.042702 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.043293 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.043428 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.044823 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.044845 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.044852 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:19.044874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.044910 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:19.044928 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.045018 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.045254 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.045356 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:34:19.046666 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.035767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46215 (local address 127.0.97.2:47461)
0620 18:34:19.036741 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.036777 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.038573 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.039239 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.039246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.039331 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:19.039965 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.039985 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.040982 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.040988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.042468 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.042485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.043559 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.043568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.043787 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.044498 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:19.044501 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:19.044543 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:19.044620 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:19.045015 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:19.045027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:19.045037 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.045231 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.045615 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":301,"threads_started":1}
I20260620 18:34:19.048157 18397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46215
I20260620 18:34:19.048456 18397 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:19.049295 18397 heartbeater.cc:511] Master 127.0.97.62:46215 requested a full tablet report, sending...
I20260620 18:34:19.050721 18080 ts_manager.cc:194] Registered new tserver with Master: 2fe2edf5a6d54159bce7f1376938b756 (127.0.97.2:44223)
I20260620 18:34:19.051976 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:47461
W20260620 18:34:19.367381 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:19.367741 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:19.367919 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:19.375197 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:19.375330 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:19.375504 18400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:19.384521 18400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:19.385735 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:19.387542 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:19.400959 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:19.400904 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:19.401001 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:19.402050 18400 server_base.cc:1034] running on GCE node
I20260620 18:34:19.402647 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:19.403584 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:19.404798 18400 hybrid_clock.cc:648] HybridClock initialized: now 1781980459404749 us; error 48 us; skew 500 ppm
I20260620 18:34:19.405067 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:19.407282 18400 webserver.cc:460] Webserver started at http://127.0.97.3:36027/ using document root <none> and password file <none>
I20260620 18:34:19.408001 18400 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:19.408088 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:19.408394 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:19.411144 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "16697b9b3d8d4f888b7b922da5bf29cb"
format_stamp: "Formatted at 2026-06-20 18:34:19 on dist-test-slave-mcnm"
server_key: "058b01c9cc58b177982b28b44d123618"
server_key_iv: "c857036596c11ef27f4e9cb0faa4fbf0"
server_key_version: "encryptionkey@0"
I20260620 18:34:19.411942 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "16697b9b3d8d4f888b7b922da5bf29cb"
format_stamp: "Formatted at 2026-06-20 18:34:19 on dist-test-slave-mcnm"
server_key: "058b01c9cc58b177982b28b44d123618"
server_key_iv: "c857036596c11ef27f4e9cb0faa4fbf0"
server_key_version: "encryptionkey@0"
I20260620 18:34:19.416894 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:34:19.420496 18414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:19.421947 18400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:19.422115 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "16697b9b3d8d4f888b7b922da5bf29cb"
format_stamp: "Formatted at 2026-06-20 18:34:19 on dist-test-slave-mcnm"
server_key: "058b01c9cc58b177982b28b44d123618"
server_key_iv: "c857036596c11ef27f4e9cb0faa4fbf0"
server_key_version: "encryptionkey@0"
I20260620 18:34:19.422367 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:19.460554 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:19.461438 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:19.461760 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:19.462988 18400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:19.464679 18400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:19.464767 18400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:19.464928 18400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:19.465018 18400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:19.499600 18400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36715
I20260620 18:34:19.499712 18526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36715 every 8 connection(s)
I20260620 18:34:19.501014 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:19.510318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18400
I20260620 18:34:19.510504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:19.511032 388 external_mini_cluster.cc:1428] Setting key 2fa12be3e6729b5db201029e67381c32
I20260620 18:34:19.514319 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.503593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46969 (local address 127.0.97.62:46215)
0620 18:34:19.503797 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.503801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.505203 (+ 1402us) server_negotiation.cc:407] Connection header received
0620 18:34:19.507047 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.507053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.507141 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.507258 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.508400 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.509152 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.511194 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.511792 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.511906 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.513451 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.513458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.513462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:19.513483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.513519 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:19.513539 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.513633 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.513952 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.514051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:34:19.515326 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.504147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46215 (local address 127.0.97.3:46969)
0620 18:34:19.505034 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.505075 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.506765 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.507487 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.507496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.507643 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:19.508232 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.508249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.509314 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.509318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.511008 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.511027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.512021 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.512027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.512288 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.513091 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:19.513095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:19.513146 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:19.513253 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:19.513667 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:19.513676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:19.513682 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.513907 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.514286 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":288,"threads_started":1}
I20260620 18:34:19.516641 18527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46215
I20260620 18:34:19.516969 18527 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:19.517695 18527 heartbeater.cc:511] Master 127.0.97.62:46215 requested a full tablet report, sending...
I20260620 18:34:19.518981 18080 ts_manager.cc:194] Registered new tserver with Master: 16697b9b3d8d4f888b7b922da5bf29cb (127.0.97.3:36715)
I20260620 18:34:19.519819 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46969
I20260620 18:34:19.526407 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:19.536244 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.530595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.97.62:46215)
0620 18:34:19.530815 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.530819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.531245 (+ 426us) server_negotiation.cc:407] Connection header received
0620 18:34:19.531379 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.531383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.531459 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.531609 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.532045 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.532738 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.534050 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.534459 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.534553 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.535653 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.535658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.535661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:19.535679 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.535709 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:19.535720 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.535805 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.535948 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.536026 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:34:19.540081 18080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:19.546799 18080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:19.570458 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.558286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.97.1:35651)
0620 18:34:19.558869 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.558879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.560437 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 18:34:19.560877 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.560886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.561134 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.561318 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:19.562430 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.564119 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.566893 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.568477 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.568631 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.568894 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.569791 (+ 897us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.569962 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":636,"threads_started":1}
I20260620 18:34:19.570506 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.558397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44223 (local address 127.0.0.1:49676)
0620 18:34:19.561305 (+ 2908us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.561330 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.561449 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.562486 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.562492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.562521 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:19.562963 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.562974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.565076 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.565081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.567465 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.567480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.569621 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.569627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.569829 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.569862 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.569948 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.570140 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2753,"mutex_wait_us":94,"thread_start_us":2619,"threads_started":1}
I20260620 18:34:19.571089 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.560482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.0.97.3:36715)
0620 18:34:19.561964 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.561972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.562590 (+ 618us) server_negotiation.cc:407] Connection header received
0620 18:34:19.562834 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.562841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.563039 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.563210 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:19.563800 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.565333 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.567916 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.569337 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.569545 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.569822 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.570598 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.570767 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1164,"threads_started":1}
I20260620 18:34:19.571415 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.558709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.97.2:44223)
0620 18:34:19.559277 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.559285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.561601 (+ 2316us) server_negotiation.cc:407] Connection header received
0620 18:34:19.561721 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.561728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.562015 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.562220 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:19.563126 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.564905 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.567732 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.569152 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.569278 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.569483 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.570927 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.571082 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":17664,"thread_start_us":383,"threads_started":1}
I20260620 18:34:19.571736 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.558061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35651 (local address 127.0.0.1:52680)
0620 18:34:19.560280 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.560327 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.560588 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.561681 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.561688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.561718 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:19.562207 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.562221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.564301 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.564307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.566683 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.566697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.569257 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.569266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.569459 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.569511 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.571049 (+ 1538us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.571384 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1962,"mutex_wait_us":1,"thread_start_us":1832,"threads_started":1}
I20260620 18:34:19.572679 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.559954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36715 (local address 127.0.0.1:48694)
0620 18:34:19.562469 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.562485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.562658 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.563216 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.563220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.563249 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:19.563630 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.563644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.565500 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.565505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.567679 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.567693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.569940 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.569945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.570141 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.570176 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.570259 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.570385 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2397,"mutex_wait_us":4,"thread_start_us":2172,"threads_started":1}
I20260620 18:34:19.575157 18201 tablet_service.cc:1467] Processing CreateTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 (DEFAULT_TABLE table=test-table [id=a61f94b76e134a03a40df13322c4b494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:19.575157 18332 tablet_service.cc:1467] Processing CreateTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 (DEFAULT_TABLE table=test-table [id=a61f94b76e134a03a40df13322c4b494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:19.575839 18462 tablet_service.cc:1467] Processing CreateTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 (DEFAULT_TABLE table=test-table [id=a61f94b76e134a03a40df13322c4b494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:19.576584 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6a7290fc2e45499a69bd61bd2e76a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:19.576587 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6a7290fc2e45499a69bd61bd2e76a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:19.577092 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6a7290fc2e45499a69bd61bd2e76a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:19.589064 18545 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Bootstrap starting.
I20260620 18:34:19.591089 18547 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Bootstrap starting.
I20260620 18:34:19.591894 18546 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Bootstrap starting.
I20260620 18:34:19.592658 18545 tablet_bootstrap.cc:654] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:19.594118 18545 log.cc:824] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:19.595885 18546 tablet_bootstrap.cc:654] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:19.597448 18546 log.cc:824] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:19.591737 18266 heartbeater.cc:503] Master 127.0.97.62:46215 was elected leader, sending a full tablet report...
I20260620 18:34:19.600997 18546 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: No bootstrap required, opened a new log
I20260620 18:34:19.601354 18546 ts_tablet_manager.cc:1397] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:34:19.603435 18547 tablet_bootstrap.cc:654] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:19.605116 18547 log.cc:824] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:19.608192 18547 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: No bootstrap required, opened a new log
I20260620 18:34:19.608489 18547 ts_tablet_manager.cc:1397] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260620 18:34:19.608754 18545 tablet_bootstrap.cc:492] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: No bootstrap required, opened a new log
I20260620 18:34:19.609047 18545 ts_tablet_manager.cc:1397] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20260620 18:34:19.610785 18546 raft_consensus.cc:348] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.611212 18546 raft_consensus.cc:374] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:19.611361 18546 raft_consensus.cc:729] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe2edf5a6d54159bce7f1376938b756, State: Initialized, Role: FOLLOWER
I20260620 18:34:19.612000 18546 consensus_queue.cc:260] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.613107 18397 heartbeater.cc:503] Master 127.0.97.62:46215 was elected leader, sending a full tablet report...
I20260620 18:34:19.613725 18546 ts_tablet_manager.cc:1428] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260620 18:34:19.617146 18547 raft_consensus.cc:348] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.617595 18547 raft_consensus.cc:374] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:19.617699 18547 raft_consensus.cc:729] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf2f4c74d7b4d07bc04367153ddd033, State: Initialized, Role: FOLLOWER
I20260620 18:34:19.618377 18547 consensus_queue.cc:260] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.618878 18545 raft_consensus.cc:348] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.619232 18545 raft_consensus.cc:374] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:19.619334 18545 raft_consensus.cc:729] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16697b9b3d8d4f888b7b922da5bf29cb, State: Initialized, Role: FOLLOWER
I20260620 18:34:19.620003 18547 ts_tablet_manager.cc:1428] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 18:34:19.620118 18545 consensus_queue.cc:260] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.621255 18527 heartbeater.cc:503] Master 127.0.97.62:46215 was elected leader, sending a full tablet report...
I20260620 18:34:19.621840 18545 ts_tablet_manager.cc:1428] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:34:19.692596 18552 raft_consensus.cc:482] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:19.692914 18552 raft_consensus.cc:504] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.694670 18552 leader_election.cc:288] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fe2edf5a6d54159bce7f1376938b756 (127.0.97.2:44223), 16697b9b3d8d4f888b7b922da5bf29cb (127.0.97.3:36715)
I20260620 18:34:19.703750 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.695021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54695 (local address 127.0.97.2:44223)
0620 18:34:19.695278 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.695288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.695995 (+ 707us) server_negotiation.cc:407] Connection header received
0620 18:34:19.696227 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.696235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.696316 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.696421 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:19.697203 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.698292 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.701461 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.702645 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.702784 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.702889 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.703334 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.703467 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:34:19.703743 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.695327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59555 (local address 127.0.97.3:36715)
0620 18:34:19.695594 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.695603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.696264 (+ 661us) server_negotiation.cc:407] Connection header received
0620 18:34:19.696393 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.696401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.696474 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.696564 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:19.697309 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.698344 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.701421 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.702677 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.702791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.702883 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.703334 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.703467 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:34:19.703930 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.694932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36715 (local address 127.0.97.1:59555)
0620 18:34:19.696156 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.696172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.696276 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.696618 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.696627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.696657 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:19.697139 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.697153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.698479 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.698483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.701189 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.701205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.702932 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.702938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.703146 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.703192 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.703291 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.703489 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":252,"thread_start_us":582,"threads_started":1}
I20260620 18:34:19.703963 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.694896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44223 (local address 127.0.97.1:54695)
0620 18:34:19.695885 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:19.695902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:19.696075 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:19.696456 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:19.696459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:19.696476 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:19.697049 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.697065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.698478 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.698483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.701261 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:19.701270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.702913 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.702921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:19.703153 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.703199 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:34:19.703291 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:34:19.703489 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":258,"spinlock_wait_cycles":71424,"thread_start_us":871,"threads_started":1}
I20260620 18:34:19.705135 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6a7290fc2e45499a69bd61bd2e76a1" candidate_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" is_pre_election: true
I20260620 18:34:19.705242 18352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6a7290fc2e45499a69bd61bd2e76a1" candidate_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe2edf5a6d54159bce7f1376938b756" is_pre_election: true
I20260620 18:34:19.705706 18352 raft_consensus.cc:2454] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf2f4c74d7b4d07bc04367153ddd033 in term 0.
I20260620 18:34:19.705713 18482 raft_consensus.cc:2454] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf2f4c74d7b4d07bc04367153ddd033 in term 0.
I20260620 18:34:19.706439 18155 leader_election.cc:302] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16697b9b3d8d4f888b7b922da5bf29cb, 2bf2f4c74d7b4d07bc04367153ddd033, 2fe2edf5a6d54159bce7f1376938b756; no voters:
I20260620 18:34:19.706837 18552 raft_consensus.cc:2781] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:19.706995 18552 raft_consensus.cc:482] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:19.707099 18552 raft_consensus.cc:3037] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:19.709339 18552 raft_consensus.cc:504] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.709993 18552 leader_election.cc:288] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [CANDIDATE]: Term 1 election: Requested vote from peers 2fe2edf5a6d54159bce7f1376938b756 (127.0.97.2:44223), 16697b9b3d8d4f888b7b922da5bf29cb (127.0.97.3:36715)
I20260620 18:34:19.710412 18352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6a7290fc2e45499a69bd61bd2e76a1" candidate_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe2edf5a6d54159bce7f1376938b756"
I20260620 18:34:19.710527 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6a7290fc2e45499a69bd61bd2e76a1" candidate_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16697b9b3d8d4f888b7b922da5bf29cb"
I20260620 18:34:19.710614 18352 raft_consensus.cc:3037] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:19.710770 18482 raft_consensus.cc:3037] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:19.712949 18352 raft_consensus.cc:2454] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf2f4c74d7b4d07bc04367153ddd033 in term 1.
I20260620 18:34:19.713054 18482 raft_consensus.cc:2454] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf2f4c74d7b4d07bc04367153ddd033 in term 1.
I20260620 18:34:19.713392 18155 leader_election.cc:302] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf2f4c74d7b4d07bc04367153ddd033, 2fe2edf5a6d54159bce7f1376938b756; no voters:
I20260620 18:34:19.713680 18552 raft_consensus.cc:2781] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:19.714114 18552 raft_consensus.cc:686] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf2f4c74d7b4d07bc04367153ddd033, State: Running, Role: LEADER
I20260620 18:34:19.714594 18552 consensus_queue.cc:237] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } }
I20260620 18:34:19.719326 18079 catalog_manager.cc:5495] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf2f4c74d7b4d07bc04367153ddd033 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf2f4c74d7b4d07bc04367153ddd033" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 } health_report { overall_health: UNKNOWN } } }
W20260620 18:34:19.752818 18528 tablet.cc:2369] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:19.766296 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.761009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.97.1:35651)
0620 18:34:19.761195 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.761199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.761216 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:34:19.761380 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.761384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.761487 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.761609 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:19.761934 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.762694 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.764175 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.764590 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.764689 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.765249 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:19.765760 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:19.765870 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.766036 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.766097 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:19.774724 18482 raft_consensus.cc:1261] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 1 FOLLOWER]: Refusing update from remote peer 2bf2f4c74d7b4d07bc04367153ddd033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:19.774727 18352 raft_consensus.cc:1261] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 1 FOLLOWER]: Refusing update from remote peer 2bf2f4c74d7b4d07bc04367153ddd033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:19.775765 18556 consensus_queue.cc:1035] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe2edf5a6d54159bce7f1376938b756" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:19.776129 18552 consensus_queue.cc:1035] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16697b9b3d8d4f888b7b922da5bf29cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:34:19.784584 18398 tablet.cc:2369] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:19.786338 18559 mvcc.cc:205] Tried to move back new op lower bound from 7298991963227484160 to 7298991962994888704. Current Snapshot: MvccSnapshot[applied={T|T < 7298991963227484160}]
I20260620 18:34:19.789176 18561 mvcc.cc:205] Tried to move back new op lower bound from 7298991963227484160 to 7298991962994888704. Current Snapshot: MvccSnapshot[applied={T|T < 7298991963227484160}]
I20260620 18:34:19.803711 18079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:19.804016 18079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:19.807615 18079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e4d5761dd5a4397a4dc143c9600b9b7: Sending DeleteTablet for 3 replicas of tablet df6a7290fc2e45499a69bd61bd2e76a1
I20260620 18:34:19.808713 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='slave'} at 127.0.0.1:52680
I20260620 18:34:19.808786 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='slave'} at 127.0.0.1:49676
I20260620 18:34:19.808786 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet df6a7290fc2e45499a69bd61bd2e76a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='slave'} at 127.0.0.1:48694
I20260620 18:34:19.813815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18139
I20260620 18:34:19.821115 18571 tablet_replica.cc:330] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: stopping tablet replica
I20260620 18:34:19.822371 18571 raft_consensus.cc:2229] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:19.823135 18570 tablet_replica.cc:330] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: stopping tablet replica
I20260620 18:34:19.823714 18570 raft_consensus.cc:2229] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:19.824304 18570 raft_consensus.cc:2258] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.824297 18571 raft_consensus.cc:2258] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.826588 18570 ts_tablet_manager.cc:1905] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.826658 18571 ts_tablet_manager.cc:1905] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.833786 18570 ts_tablet_manager.cc:1918] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.833999 18571 ts_tablet_manager.cc:1918] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.834192 18571 log.cc:1196] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/df6a7290fc2e45499a69bd61bd2e76a1
I20260620 18:34:19.834645 18569 tablet_replica.cc:330] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: stopping tablet replica
I20260620 18:34:19.834968 18571 ts_tablet_manager.cc:1939] T df6a7290fc2e45499a69bd61bd2e76a1 P 2bf2f4c74d7b4d07bc04367153ddd033: Deleting consensus metadata
I20260620 18:34:19.835170 18569 raft_consensus.cc:2229] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:19.835785 18569 raft_consensus.cc:2258] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.836519 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.815115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.0.97.1:35651)
0620 18:34:19.815284 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.815290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.827487 (+ 12197us) server_negotiation.cc:407] Connection header received
0620 18:34:19.828005 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.828011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.828127 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.828208 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.828894 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.830131 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.832266 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.832982 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.833087 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.834666 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.834677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.834694 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:19.835600 (+ 906us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.835658 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:19.835682 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.835759 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.836038 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.836155 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:19.837136 18064 catalog_manager.cc:4841] TS 2bf2f4c74d7b4d07bc04367153ddd033 (127.0.97.1:35651): tablet df6a7290fc2e45499a69bd61bd2e76a1 (table test-table [id=a61f94b76e134a03a40df13322c4b494]) successfully deleted
I20260620 18:34:19.838210 18569 ts_tablet_manager.cc:1905] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.838538 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:19.839632 18570 log.cc:1196] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/df6a7290fc2e45499a69bd61bd2e76a1
I20260620 18:34:19.840865 18570 ts_tablet_manager.cc:1939] T df6a7290fc2e45499a69bd61bd2e76a1 P 16697b9b3d8d4f888b7b922da5bf29cb: Deleting consensus metadata
I20260620 18:34:19.843456 18067 catalog_manager.cc:4841] TS 16697b9b3d8d4f888b7b922da5bf29cb (127.0.97.3:36715): tablet df6a7290fc2e45499a69bd61bd2e76a1 (table test-table [id=a61f94b76e134a03a40df13322c4b494]) successfully deleted
I20260620 18:34:19.845783 18569 ts_tablet_manager.cc:1918] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.845944 18569 log.cc:1196] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/df6a7290fc2e45499a69bd61bd2e76a1
I20260620 18:34:19.846449 18569 ts_tablet_manager.cc:1939] T df6a7290fc2e45499a69bd61bd2e76a1 P 2fe2edf5a6d54159bce7f1376938b756: Deleting consensus metadata
I20260620 18:34:19.848280 18065 catalog_manager.cc:4841] TS 2fe2edf5a6d54159bce7f1376938b756 (127.0.97.2:44223): tablet df6a7290fc2e45499a69bd61bd2e76a1 (table test-table [id=a61f94b76e134a03a40df13322c4b494]) successfully deleted
I20260620 18:34:19.937193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18139
I20260620 18:34:19.956686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18270
I20260620 18:34:19.963979 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.957632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.97.2:44223)
0620 18:34:19.957846 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.957850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.957868 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:34:19.958009 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.958013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.958092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.958188 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.958551 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.959285 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.960807 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.961213 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.961345 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.962489 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.962498 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.962513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:19.963134 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.963184 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:19.963204 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.963303 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.963443 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.963605 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:34:19.964705 18392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:20.054617 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18270
I20260620 18:34:20.071698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18400
I20260620 18:34:20.079036 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.072486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.97.3:36715)
0620 18:34:20.072669 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.072677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.072757 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:34:20.072983 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.072988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.073085 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.073197 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.073601 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.074573 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.076077 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.076472 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.076614 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.077712 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.077722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.077738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:20.078293 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.078339 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:20.078358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.078484 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.078624 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.078738 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:34:20.079725 18522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:20.169389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18400
I20260620 18:34:20.187170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18049
I20260620 18:34:20.188470 18110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:20.296866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18049
2026-06-20T18:34:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2800 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:20 dist-test-slave-mcnm krb5kdc[18579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:20 dist-test-slave-mcnm krb5kdc[18579](info): set up 2 sockets
Jun 20 18:34:20 dist-test-slave-mcnm krb5kdc[18579](info): commencing operation
krb5kdc: starting...
W20260620 18:34:21.411700 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:21 dist-test-slave-mcnm krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:34:21.654376 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:21.981554 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:21.981820 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:21.981982 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:21.989017 18592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:21.989114 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:21.989194 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:21.989305 18592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:21.989364 18592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:21.998867 18592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:22.000229 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:22.001596 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:22.013439 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:22.014034 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:22.014076 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:22.014380 18592 server_base.cc:1034] running on GCE node
I20260620 18:34:22.015259 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:22.016777 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:22.018028 18592 hybrid_clock.cc:648] HybridClock initialized: now 1781980462017969 us; error 55 us; skew 500 ppm
I20260620 18:34:22.018393 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:22 dist-test-slave-mcnm krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:22.021504 18592 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:34:22.022974 18592 webserver.cc:460] Webserver started at http://127.0.97.62:43007/ using document root <none> and password file <none>
I20260620 18:34:22.023722 18592 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:22.023876 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:22.024186 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:22.027165 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "b5bfc4c4bab443ca9cd130aaa1db8167"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "94c2ac90b3c00848a01d9b1dc8f37777"
server_key_iv: "b62bcc7d847d8961404e4d2175bb092a"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.027956 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "b5bfc4c4bab443ca9cd130aaa1db8167"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "94c2ac90b3c00848a01d9b1dc8f37777"
server_key_iv: "b62bcc7d847d8961404e4d2175bb092a"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.033232 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:22.036715 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:22.038000 18592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:34:22.038182 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "b5bfc4c4bab443ca9cd130aaa1db8167"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "94c2ac90b3c00848a01d9b1dc8f37777"
server_key_iv: "b62bcc7d847d8961404e4d2175bb092a"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.038399 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:22.051329 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:22.055228 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:22.055558 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:22.074086 18592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38675
I20260620 18:34:22.074128 18658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38675 every 8 connection(s)
I20260620 18:34:22.075479 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:22.077133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18592
I20260620 18:34:22.077330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:22.077756 388 external_mini_cluster.cc:1428] Setting key bee886ba99ea22628a37b137e2d95d5d
I20260620 18:34:22.079627 18659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:34:22 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:22.092553 18659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: Bootstrap starting.
I20260620 18:34:22.096474 18659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.097754 18659 log.cc:824] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:22.100500 18659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: No bootstrap required, opened a new log
I20260620 18:34:22.101771 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:22.079706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.0.97.62:38675)
0620 18:34:22.080475 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:22.080491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:22.080546 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:34:22.082688 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:22.082714 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:22.083208 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:22.083659 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:22.084940 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.086929 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.088459 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.089474 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.089646 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:22.093616 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:22.093626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:22.093643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:22.093689 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:22.097613 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.098394 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.098401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.098406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.098587 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.099028 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.099035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.099043 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.099609 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:22.099985 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:22.100419 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:34:22.100840 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":316,"threads_started":1}
I20260620 18:34:22.108700 18659 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } }
I20260620 18:34:22.109071 18659 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.109177 18659 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bfc4c4bab443ca9cd130aaa1db8167, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.109712 18659 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } }
I20260620 18:34:22.109959 18659 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:22.110101 18659 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:22.110286 18659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:22.112480 18659 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } }
I20260620 18:34:22.112970 18659 leader_election.cc:302] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bfc4c4bab443ca9cd130aaa1db8167; no voters:
I20260620 18:34:22.113582 18659 leader_election.cc:288] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:22.113700 18664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:22.114753 18664 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [term 1 LEADER]: Becoming Leader. State: Replica: b5bfc4c4bab443ca9cd130aaa1db8167, State: Running, Role: LEADER
I20260620 18:34:22.115378 18664 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } }
I20260620 18:34:22.115918 18659 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:22.118093 18665 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } } }
I20260620 18:34:22.118481 18665 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:22.119056 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bfc4c4bab443ca9cd130aaa1db8167. Latest consensus state: current_term: 1 leader_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bfc4c4bab443ca9cd130aaa1db8167" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38675 } } }
I20260620 18:34:22.119364 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:22.119387 18673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:22.127532 18673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:22.137903 18673 catalog_manager.cc:1347] Generated new cluster ID: 03d641658d9a4df6927007141ef99414
I20260620 18:34:22.138010 18673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:22.149555 18673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:22.150820 18673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:22.156313 18673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: Generated new TSK 0
I20260620 18:34:22.157060 18673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:34:22.219384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:22.550839 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:22.551175 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:22.551347 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:22.558815 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:22.558965 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:22.559151 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:22.568907 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:22.570195 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:22.571568 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:22.584467 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:22.584545 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:22.584681 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:22.585707 18686 server_base.cc:1034] running on GCE node
I20260620 18:34:22.586294 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:22.587237 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:22.588438 18686 hybrid_clock.cc:648] HybridClock initialized: now 1781980462588374 us; error 57 us; skew 500 ppm
I20260620 18:34:22.588747 18686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:22 dist-test-slave-mcnm krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:22.591924 18686 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:34:22.593458 18686 webserver.cc:460] Webserver started at http://127.0.97.1:36567/ using document root <none> and password file <none>
I20260620 18:34:22.594137 18686 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:22.594265 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:22.594579 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:22.597450 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b7c0dd9f643444daf52125a1ed27592"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "d1a6bfa032198807a398f9f539102bfb"
server_key_iv: "f626e066c6d626db2e80293c60d8b340"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.598248 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7c0dd9f643444daf52125a1ed27592"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "d1a6bfa032198807a398f9f539102bfb"
server_key_iv: "f626e066c6d626db2e80293c60d8b340"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.603477 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:34:22.607012 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:22.608466 18686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:22.608644 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "8b7c0dd9f643444daf52125a1ed27592"
format_stamp: "Formatted at 2026-06-20 18:34:22 on dist-test-slave-mcnm"
server_key: "d1a6bfa032198807a398f9f539102bfb"
server_key_iv: "f626e066c6d626db2e80293c60d8b340"
server_key_version: "encryptionkey@0"
I20260620 18:34:22.608822 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:22.624625 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:22.628829 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:22.629123 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:22.630188 18686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:22.631476 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:22.631615 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:22.631780 18686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:22.631850 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:22.664933 18686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44483
I20260620 18:34:22.664976 18813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44483 every 8 connection(s)
I20260620 18:34:22.666200 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:22.675583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18686
I20260620 18:34:22.675772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:22.676236 388 external_mini_cluster.cc:1428] Setting key fb8c958a1833a22d89b2d3df133a01d1
Jun 20 18:34:22 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:22.688050 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:22.668727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59125 (local address 127.0.97.62:38675)
0620 18:34:22.669188 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:22.669194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:22.670473 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 18:34:22.672681 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:22.672686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:22.672753 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:22.672849 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:22.674324 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.675077 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.676844 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.677358 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.677475 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:22.681315 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:22.681324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:22.681331 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:22.681374 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:22.684848 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.685542 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.685550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.685556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.685704 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.686103 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.686110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.686116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.686507 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:22.686662 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:22.687306 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 18:34:22.687583 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260620 18:34:22.688866 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:22.669255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38675 (local address 127.0.97.1:59125)
0620 18:34:22.670291 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:22.670331 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:22.672412 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:22.673101 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:22.673109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:22.673608 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:22.674163 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:22.674181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.675218 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.675222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:22.676664 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:22.676699 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.677600 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.677608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:22.677879 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:22.678669 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:22.678691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:22.681095 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:22.684982 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:22.684988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:22.685001 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:22.685409 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:22.685809 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:22.685812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:22.685814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:22.685988 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:22.686625 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:22.686630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:22.686926 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 18:34:22.687134 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 18:34:22.687571 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":271,"threads_started":1}
I20260620 18:34:22.690378 18814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38675
I20260620 18:34:22.690671 18814 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:22.691380 18814 heartbeater.cc:511] Master 127.0.97.62:38675 requested a full tablet report, sending...
I20260620 18:34:22.693498 18624 ts_manager.cc:194] Registered new tserver with Master: 8b7c0dd9f643444daf52125a1ed27592 (127.0.97.1:44483)
I20260620 18:34:22.694890 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59125
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:34:22.731339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:23.054745 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:23.055083 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:23.055258 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:23.062603 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:23.062740 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:23.062927 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:23.072820 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:23.074221 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:23.076474 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:23.097131 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:23.097184 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:23.097888 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:23.098980 18820 server_base.cc:1034] running on GCE node
I20260620 18:34:23.099785 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:23.101080 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:23.102330 18820 hybrid_clock.cc:648] HybridClock initialized: now 1781980463102259 us; error 61 us; skew 500 ppm
I20260620 18:34:23.102718 18820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:23 dist-test-slave-mcnm krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:23.107228 18820 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:34:23.109190 18820 webserver.cc:460] Webserver started at http://127.0.97.2:43747/ using document root <none> and password file <none>
I20260620 18:34:23.110005 18820 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:23.110111 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:23.110375 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:23.114050 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "a12d3c952a384e17af7a96066cf19927"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "1bbf4fa93baadee2534101f17c65734a"
server_key_iv: "2b76270b69eb52c9a2b489e7d403fb01"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.115265 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a12d3c952a384e17af7a96066cf19927"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "1bbf4fa93baadee2534101f17c65734a"
server_key_iv: "2b76270b69eb52c9a2b489e7d403fb01"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.123139 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 18:34:23.127223 18835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.129002 18820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 18:34:23.129166 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "a12d3c952a384e17af7a96066cf19927"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "1bbf4fa93baadee2534101f17c65734a"
server_key_iv: "2b76270b69eb52c9a2b489e7d403fb01"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.129379 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:23.153427 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:23.157724 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:23.158025 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:23.158991 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:23.160405 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:23.160521 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.160660 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:23.160730 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.196637 18820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39781
I20260620 18:34:23.196679 18947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39781 every 8 connection(s)
I20260620 18:34:23.197962 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:23.204941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18820
I20260620 18:34:23.205160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:23.205605 388 external_mini_cluster.cc:1428] Setting key 319565831180f4c8796b2bdb564f5960
Jun 20 18:34:23 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:23.220129 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.200673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40447 (local address 127.0.97.62:38675)
0620 18:34:23.201126 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.201131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.202097 (+ 966us) server_negotiation.cc:407] Connection header received
0620 18:34:23.204037 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.204043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.204130 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.204248 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:23.205895 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.206893 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.208793 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.209616 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.209771 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.213860 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:23.213865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:23.213868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:23.213900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:23.216840 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.217576 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.217582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.217586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.217722 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.218238 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.218242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.218244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.218528 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:23.218708 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.219632 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.219852 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260620 18:34:23.221611 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.201231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38675 (local address 127.0.97.2:40447)
0620 18:34:23.201925 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.201969 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.203765 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.204487 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.204503 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.205025 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:23.205672 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.205716 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.207072 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.207077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.208642 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.208654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.209905 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.209915 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.210206 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.210906 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:23.210928 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:23.213624 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:23.216982 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:23.216991 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:23.217009 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:23.217446 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:23.217836 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:23.217843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:23.217849 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:23.218112 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:23.218696 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:23.218707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:23.219189 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.219485 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.220103 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
I20260620 18:34:23.223115 18948 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38675
I20260620 18:34:23.223425 18948 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:23.224162 18948 heartbeater.cc:511] Master 127.0.97.62:38675 requested a full tablet report, sending...
I20260620 18:34:23.225682 18624 ts_manager.cc:194] Registered new tserver with Master: a12d3c952a384e17af7a96066cf19927 (127.0.97.2:39781)
I20260620 18:34:23.226496 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40447
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:34:23.262669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:23.586611 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:23.586956 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:23.587131 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:23.594475 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:23.594641 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:23.594832 18954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:23.604715 18954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:23.606061 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:23.607443 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:23.621284 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:23.621182 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:23.621232 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:23.621935 18954 server_base.cc:1034] running on GCE node
I20260620 18:34:23.622599 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:23.623658 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:23.624976 18954 hybrid_clock.cc:648] HybridClock initialized: now 1781980463624835 us; error 89 us; skew 500 ppm
I20260620 18:34:23.625344 18954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:23 dist-test-slave-mcnm krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:23.628811 18954 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:34:23.630292 18954 webserver.cc:460] Webserver started at http://127.0.97.3:43795/ using document root <none> and password file <none>
I20260620 18:34:23.631048 18954 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:23.631206 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:23.631595 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:23.634860 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "623aab5389f1458a8b360e4344889033"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "6d722c24e68971a2f422038042a367c7"
server_key_iv: "28ac0e1b9dc8996e66e5aa5edeac9409"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.635763 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "623aab5389f1458a8b360e4344889033"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "6d722c24e68971a2f422038042a367c7"
server_key_iv: "28ac0e1b9dc8996e66e5aa5edeac9409"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.641176 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:23.645123 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.646500 18954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:34:23.646688 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "623aab5389f1458a8b360e4344889033"
format_stamp: "Formatted at 2026-06-20 18:34:23 on dist-test-slave-mcnm"
server_key: "6d722c24e68971a2f422038042a367c7"
server_key_iv: "28ac0e1b9dc8996e66e5aa5edeac9409"
server_key_version: "encryptionkey@0"
I20260620 18:34:23.646903 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:23.661661 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:23.666091 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:23.666417 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:23.667734 18954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:23.669243 18954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:23.669322 18954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.669486 18954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:23.669539 18954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:23.697718 18814 heartbeater.cc:503] Master 127.0.97.62:38675 was elected leader, sending a full tablet report...
I20260620 18:34:23.703608 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45703
I20260620 18:34:23.703650 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45703 every 8 connection(s)
I20260620 18:34:23.704892 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:23.705206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 18954
I20260620 18:34:23.705384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:23.705931 388 external_mini_cluster.cc:1428] Setting key 4758060ecca35b88de0829aa68894ded
Jun 20 18:34:23 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:23.725661 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.707721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43475 (local address 127.0.97.62:38675)
0620 18:34:23.707897 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.707901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.709334 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 18:34:23.711198 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.711202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.711269 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.711353 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:23.712938 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.713736 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.715294 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.715794 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.715925 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.719780 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:23.719789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:23.719797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:23.719847 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:23.722819 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.723576 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.723584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.723592 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.723734 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.724130 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.724134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.724137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.724401 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:23.724508 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.725240 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.725426 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:34:23.726717 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.708350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38675 (local address 127.0.97.3:43475)
0620 18:34:23.709181 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.709223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.710871 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.711628 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.711639 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.712195 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:23.712762 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.712779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.713886 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.713892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.715147 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.715159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.716027 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.716034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.716286 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.716894 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:23.716915 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:23.719541 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:23.722952 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:23.722960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:23.722974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:23.723412 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:23.723830 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:23.723834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:23.723836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:23.724024 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:23.724504 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:23.724510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:23.724870 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.725107 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.725512 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":382,"threads_started":1}
I20260620 18:34:23.728216 19082 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38675
I20260620 18:34:23.728535 19082 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:23.729285 19082 heartbeater.cc:511] Master 127.0.97.62:38675 requested a full tablet report, sending...
I20260620 18:34:23.731000 18624 ts_manager.cc:194] Registered new tserver with Master: 623aab5389f1458a8b360e4344889033 (127.0.97.3:45703)
I20260620 18:34:23.731985 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43475
I20260620 18:34:23.734138 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:23.752139 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.738403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.0.97.62:38675)
0620 18:34:23.738576 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.738580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.739112 (+ 532us) server_negotiation.cc:407] Connection header received
0620 18:34:23.739262 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.739266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.739321 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.739400 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:23.740992 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.741809 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.743375 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.743826 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.743923 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.746588 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:23.746593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:23.746596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:23.746627 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:23.749401 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.750243 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.750247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.750249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.750360 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:23.750744 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:23.750748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:23.750750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:23.751061 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:23.751366 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.751762 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.751905 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:34:23.756110 18623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:23.762980 18623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:23.785562 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.775642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.97.1:44483)
0620 18:34:23.777140 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.777149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.777179 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:34:23.777281 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.777290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.777566 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.777776 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:23.778709 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.780447 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.782614 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.784192 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.784331 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.784561 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.785064 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.785242 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":1138,"threads_started":1}
I20260620 18:34:23.785698 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.775314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44483 (local address 127.0.0.1:33326)
0620 18:34:23.776353 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.776413 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.776606 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.777813 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.777820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.777852 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:23.778455 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.778469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.780606 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.780612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.782450 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.782462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.784478 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.784486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.784760 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.784862 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.785003 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.785229 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":144,"spinlock_wait_cycles":2816,"thread_start_us":609,"threads_started":1}
I20260620 18:34:23.786535 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.775374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39781 (local address 127.0.0.1:53862)
0620 18:34:23.777766 (+ 2392us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.777785 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.777910 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.778618 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.778625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.778652 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:23.779077 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.779099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.781094 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.781099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.783115 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.783127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.785231 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.785237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.785414 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.785448 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.785520 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.786151 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"spinlock_wait_cycles":768,"thread_start_us":2100,"threads_started":1}
I20260620 18:34:23.787371 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.777837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45703 (local address 127.0.0.1:37976)
0620 18:34:23.778377 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.778397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.778525 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.780075 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.780080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.780101 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:23.780497 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.780510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.782743 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.782748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.784812 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.784824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.786500 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.786506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.786709 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.786738 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.786822 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.786944 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":22,"thread_start_us":791,"threads_started":1}
I20260620 18:34:23.787424 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.778440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.0.97.3:45703)
0620 18:34:23.779259 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.779265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.779288 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:34:23.779377 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.779398 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.779875 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.780697 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:23.780962 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.782597 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.784965 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.786287 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.786435 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.786663 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.786889 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.787042 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":481,"threads_started":1}
I20260620 18:34:23.788194 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.775926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.0.97.2:39781)
0620 18:34:23.777560 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.777568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.778051 (+ 483us) server_negotiation.cc:407] Connection header received
0620 18:34:23.778175 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.778185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.778420 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.779173 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:23.779419 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.780950 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.783551 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.784998 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.786947 (+ 1949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.787183 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.787250 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.787412 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1340,"threads_started":1}
I20260620 18:34:23.789960 18749 tablet_service.cc:1467] Processing CreateTablet for tablet a1becdeb54754b9086332e5272ecba46 (DEFAULT_TABLE table=test-table [id=b220077c901a4616840dff440829bfbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:23.791213 18749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1becdeb54754b9086332e5272ecba46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:23.791132 19017 tablet_service.cc:1467] Processing CreateTablet for tablet a1becdeb54754b9086332e5272ecba46 (DEFAULT_TABLE table=test-table [id=b220077c901a4616840dff440829bfbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:23.791680 18883 tablet_service.cc:1467] Processing CreateTablet for tablet a1becdeb54754b9086332e5272ecba46 (DEFAULT_TABLE table=test-table [id=b220077c901a4616840dff440829bfbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:23.792594 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1becdeb54754b9086332e5272ecba46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:23.793210 18883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1becdeb54754b9086332e5272ecba46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:23.802676 19100 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Bootstrap starting.
I20260620 18:34:23.806025 19100 tablet_bootstrap.cc:654] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:23.807327 19100 log.cc:824] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:23.808298 19101 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Bootstrap starting.
I20260620 18:34:23.811659 19102 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Bootstrap starting.
I20260620 18:34:23.812999 19101 tablet_bootstrap.cc:654] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:23.814239 19101 log.cc:824] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:23.814985 19102 tablet_bootstrap.cc:654] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:23.815578 19100 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: No bootstrap required, opened a new log
I20260620 18:34:23.815872 19100 ts_tablet_manager.cc:1397] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260620 18:34:23.816226 19102 log.cc:824] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:23.818089 19101 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: No bootstrap required, opened a new log
I20260620 18:34:23.818363 19101 ts_tablet_manager.cc:1397] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:34:23.819341 19102 tablet_bootstrap.cc:492] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: No bootstrap required, opened a new log
I20260620 18:34:23.819684 19102 ts_tablet_manager.cc:1397] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 18:34:23.823846 19100 raft_consensus.cc:348] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.824213 19100 raft_consensus.cc:374] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:23.824316 19100 raft_consensus.cc:729] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7c0dd9f643444daf52125a1ed27592, State: Initialized, Role: FOLLOWER
I20260620 18:34:23.824935 19100 consensus_queue.cc:260] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.826761 19100 ts_tablet_manager.cc:1428] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 18:34:23.829264 19101 raft_consensus.cc:348] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.829721 19101 raft_consensus.cc:374] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:23.829824 19101 raft_consensus.cc:729] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623aab5389f1458a8b360e4344889033, State: Initialized, Role: FOLLOWER
I20260620 18:34:23.830453 19101 consensus_queue.cc:260] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.830355 19102 raft_consensus.cc:348] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.830781 19102 raft_consensus.cc:374] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:23.830883 19102 raft_consensus.cc:729] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12d3c952a384e17af7a96066cf19927, State: Initialized, Role: FOLLOWER
I20260620 18:34:23.831604 19082 heartbeater.cc:503] Master 127.0.97.62:38675 was elected leader, sending a full tablet report...
I20260620 18:34:23.832295 19101 ts_tablet_manager.cc:1428] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 18:34:23.832659 19102 consensus_queue.cc:260] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.834556 18948 heartbeater.cc:503] Master 127.0.97.62:38675 was elected leader, sending a full tablet report...
I20260620 18:34:23.835289 19102 ts_tablet_manager.cc:1428] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 18:34:23.878273 19107 raft_consensus.cc:482] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:23.878548 19107 raft_consensus.cc:504] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.880303 19107 leader_election.cc:288] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b7c0dd9f643444daf52125a1ed27592 (127.0.97.1:44483), a12d3c952a384e17af7a96066cf19927 (127.0.97.2:39781)
I20260620 18:34:23.886219 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.880878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55177 (local address 127.0.97.1:44483)
0620 18:34:23.881080 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.881088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.881321 (+ 233us) server_negotiation.cc:407] Connection header received
0620 18:34:23.881552 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.881558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.881628 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.881729 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:23.882133 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.882852 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.884527 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.885323 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.885474 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.885574 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.885832 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.885960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:34:23.886237 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.880914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44483 (local address 127.0.97.3:55177)
0620 18:34:23.881223 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.881239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.881409 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.881759 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.881764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.881797 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:23.881997 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.882006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.882988 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.882993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.884374 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.884386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.885485 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.885491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.885668 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.885693 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.885806 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.885936 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":22}
I20260620 18:34:23.886953 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.880831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54939 (local address 127.0.97.2:39781)
0620 18:34:23.881049 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.881056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.881903 (+ 847us) server_negotiation.cc:407] Connection header received
0620 18:34:23.882115 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.882125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.882188 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.882291 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:23.882794 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.883472 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.885189 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.885946 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.886111 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.886207 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.886602 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.886759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:34:23.887224 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.880972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39781 (local address 127.0.97.3:54939)
0620 18:34:23.881784 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.881799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.881959 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.882304 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.882308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.882330 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:23.882622 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.882630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.883637 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.883645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.885026 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.885045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.886175 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.886180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.886383 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.886432 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.886557 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.886776 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":20,"thread_start_us":491,"threads_started":1}
I20260620 18:34:23.887332 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1becdeb54754b9086332e5272ecba46" candidate_uuid: "623aab5389f1458a8b360e4344889033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7c0dd9f643444daf52125a1ed27592" is_pre_election: true
I20260620 18:34:23.887977 18769 raft_consensus.cc:2454] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623aab5389f1458a8b360e4344889033 in term 0.
I20260620 18:34:23.888465 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1becdeb54754b9086332e5272ecba46" candidate_uuid: "623aab5389f1458a8b360e4344889033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12d3c952a384e17af7a96066cf19927" is_pre_election: true
I20260620 18:34:23.888691 18971 leader_election.cc:302] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623aab5389f1458a8b360e4344889033, 8b7c0dd9f643444daf52125a1ed27592; no voters:
I20260620 18:34:23.889035 18903 raft_consensus.cc:2454] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623aab5389f1458a8b360e4344889033 in term 0.
I20260620 18:34:23.889178 19107 raft_consensus.cc:2781] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:23.889362 19107 raft_consensus.cc:482] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:23.889482 19107 raft_consensus.cc:3037] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:23.891861 19107 raft_consensus.cc:504] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.892470 19107 leader_election.cc:288] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [CANDIDATE]: Term 1 election: Requested vote from peers 8b7c0dd9f643444daf52125a1ed27592 (127.0.97.1:44483), a12d3c952a384e17af7a96066cf19927 (127.0.97.2:39781)
I20260620 18:34:23.892805 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1becdeb54754b9086332e5272ecba46" candidate_uuid: "623aab5389f1458a8b360e4344889033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7c0dd9f643444daf52125a1ed27592"
I20260620 18:34:23.892894 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1becdeb54754b9086332e5272ecba46" candidate_uuid: "623aab5389f1458a8b360e4344889033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12d3c952a384e17af7a96066cf19927"
I20260620 18:34:23.893007 18769 raft_consensus.cc:3037] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:23.893126 18903 raft_consensus.cc:3037] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:23.895195 18769 raft_consensus.cc:2454] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623aab5389f1458a8b360e4344889033 in term 1.
I20260620 18:34:23.895361 18903 raft_consensus.cc:2454] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623aab5389f1458a8b360e4344889033 in term 1.
I20260620 18:34:23.895623 18971 leader_election.cc:302] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623aab5389f1458a8b360e4344889033, 8b7c0dd9f643444daf52125a1ed27592; no voters:
I20260620 18:34:23.895913 19107 raft_consensus.cc:2781] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:23.896327 19107 raft_consensus.cc:686] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 1 LEADER]: Becoming Leader. State: Replica: 623aab5389f1458a8b360e4344889033, State: Running, Role: LEADER
I20260620 18:34:23.896782 19107 consensus_queue.cc:237] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } }
I20260620 18:34:23.901430 18624 catalog_manager.cc:5495] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 623aab5389f1458a8b360e4344889033 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "623aab5389f1458a8b360e4344889033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "623aab5389f1458a8b360e4344889033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45703 } health_report { overall_health: HEALTHY } } }
W20260620 18:34:23.918560 18815 tablet.cc:2369] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:23.942586 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.936876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.0.97.3:45703)
0620 18:34:23.937062 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.937067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.937153 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:34:23.937335 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.937339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.937417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.937548 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:23.937952 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.938667 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.940173 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.940563 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.940658 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.941275 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:23.941886 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:23.942076 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.942253 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.942299 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 18:34:23.950026 18949 tablet.cc:2369] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:23.951152 18903 raft_consensus.cc:1261] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 1 FOLLOWER]: Refusing update from remote peer 623aab5389f1458a8b360e4344889033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:23.951216 18769 raft_consensus.cc:1261] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 1 FOLLOWER]: Refusing update from remote peer 623aab5389f1458a8b360e4344889033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:23.952219 19110 consensus_queue.cc:1035] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12d3c952a384e17af7a96066cf19927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:23.952585 19107 consensus_queue.cc:1035] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7c0dd9f643444daf52125a1ed27592" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:34:23.956693 19083 tablet.cc:2369] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:23.976136 18623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:23.976413 18623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:23.979774 18623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5bfc4c4bab443ca9cd130aaa1db8167: Sending DeleteTablet for 3 replicas of tablet a1becdeb54754b9086332e5272ecba46
I20260620 18:34:23.980934 18749 tablet_service.cc:1514] Processing DeleteTablet for tablet a1becdeb54754b9086332e5272ecba46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33326
I20260620 18:34:23.980998 18883 tablet_service.cc:1514] Processing DeleteTablet for tablet a1becdeb54754b9086332e5272ecba46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53862
I20260620 18:34:23.981424 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet a1becdeb54754b9086332e5272ecba46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37976
I20260620 18:34:23.986181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18686
I20260620 18:34:23.995086 19123 tablet_replica.cc:330] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: stopping tablet replica
I20260620 18:34:23.995747 19122 tablet_replica.cc:330] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: stopping tablet replica
I20260620 18:34:23.995765 19124 tablet_replica.cc:330] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: stopping tablet replica
I20260620 18:34:23.996222 19123 raft_consensus.cc:2229] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:23.996208 19124 raft_consensus.cc:2229] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:23.996215 19122 raft_consensus.cc:2229] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:23.996744 19122 raft_consensus.cc:2258] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:23.996742 19123 raft_consensus.cc:2258] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:23.996867 19124 raft_consensus.cc:2258] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:23.999221 19123 ts_tablet_manager.cc:1905] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:23.999343 19122 ts_tablet_manager.cc:1905] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:34:24 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:34:24.006489 19122 ts_tablet_manager.cc:1918] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:24.006671 19122 log.cc:1196] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/a1becdeb54754b9086332e5272ecba46
I20260620 18:34:24.006696 19123 ts_tablet_manager.cc:1918] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:24.006857 19123 log.cc:1196] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/a1becdeb54754b9086332e5272ecba46
I20260620 18:34:24.007200 19122 ts_tablet_manager.cc:1939] T a1becdeb54754b9086332e5272ecba46 P a12d3c952a384e17af7a96066cf19927: Deleting consensus metadata
I20260620 18:34:24.007449 19123 ts_tablet_manager.cc:1939] T a1becdeb54754b9086332e5272ecba46 P 8b7c0dd9f643444daf52125a1ed27592: Deleting consensus metadata
I20260620 18:34:24.009114 19124 ts_tablet_manager.cc:1905] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:24.009650 18609 catalog_manager.cc:4841] TS 8b7c0dd9f643444daf52125a1ed27592 (127.0.97.1:44483): tablet a1becdeb54754b9086332e5272ecba46 (table test-table [id=b220077c901a4616840dff440829bfbf]) successfully deleted
I20260620 18:34:24.011622 18611 catalog_manager.cc:4841] TS a12d3c952a384e17af7a96066cf19927 (127.0.97.2:39781): tablet a1becdeb54754b9086332e5272ecba46 (table test-table [id=b220077c901a4616840dff440829bfbf]) successfully deleted
I20260620 18:34:24.014689 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.988367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.97.1:44483)
0620 18:34:23.988549 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.988556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.991926 (+ 3370us) server_negotiation.cc:407] Connection header received
0620 18:34:23.992151 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.992158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.992250 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.992355 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:23.997316 (+ 4961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.998436 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.000537 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:24.001241 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.001337 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:24.007095 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:24.007112 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:24.007133 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:24.007186 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:24.011203 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.012252 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.012264 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.012268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.012456 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.012947 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.012952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.012956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.013365 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:24.013779 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:24.013981 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:34:24.014196 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:24.015400 18809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:24.017928 19124 ts_tablet_manager.cc:1918] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:24.018249 19124 log.cc:1196] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/a1becdeb54754b9086332e5272ecba46
I20260620 18:34:24.019065 19124 ts_tablet_manager.cc:1939] T a1becdeb54754b9086332e5272ecba46 P 623aab5389f1458a8b360e4344889033: Deleting consensus metadata
I20260620 18:34:24.020937 18611 catalog_manager.cc:4841] TS 623aab5389f1458a8b360e4344889033 (127.0.97.3:45703): tablet a1becdeb54754b9086332e5272ecba46 (table test-table [id=b220077c901a4616840dff440829bfbf]) successfully deleted
I20260620 18:34:24.114916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18686
I20260620 18:34:24.133689 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18820
Jun 20 18:34:24 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:34:24.151712 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:24.134575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.97.2:39781)
0620 18:34:24.134770 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:24.134777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:24.134799 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:34:24.134925 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:24.134931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:24.135025 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:24.135136 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:24.136495 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:24.137627 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.138984 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:24.139650 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.139784 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:24.144897 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:24.144916 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:24.144936 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:24.144987 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:24.148696 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.149414 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.149423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.149427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.149581 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.149953 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.149959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.149962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.150499 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:24.150777 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:24.151084 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 18:34:24.151317 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:34:24.152400 18943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:24.244398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18820
I20260620 18:34:24.268704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18954
Jun 20 18:34:24 dist-test-slave-mcnm krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:34:24.289428 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:24.269489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.0.97.3:45703)
0620 18:34:24.269664 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:24.269668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:24.269750 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:34:24.269946 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:24.269950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:24.270015 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:24.270110 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:24.271991 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:24.272918 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.274929 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:24.275692 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:24.275827 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:24.281772 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:24.281787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:24.281807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:24.281848 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:24.286149 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.287004 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.287015 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.287019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.287227 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:24.287657 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:24.287662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:24.287666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:24.288137 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:24.288492 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:24.288730 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 18:34:24.288977 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:34:24.290225 19077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:24.414394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18954
I20260620 18:34:24.435905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 18592
I20260620 18:34:24.437067 18654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:24.549330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 18592
2026-06-20T18:34:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:24Z Disabled control of system clock
I20260620 18:34:24.625716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41251
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:24.959034 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:24.959306 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:24.959391 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:24.966542 19133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:24.966679 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:24.966732 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:24.966789 19133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:24.966840 19133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:24.976004 19133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:24.977265 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:24.979031 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:24.995648 19138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:24.995595 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:24.995674 19141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:24.996559 19133 server_base.cc:1034] running on GCE node
I20260620 18:34:24.997364 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:24.998668 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:24.999871 19133 hybrid_clock.cc:648] HybridClock initialized: now 1781980464999822 us; error 42 us; skew 500 ppm
I20260620 18:34:25.000170 19133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:25.002426 19133 webserver.cc:460] Webserver started at http://127.0.97.62:33619/ using document root <none> and password file <none>
I20260620 18:34:25.003062 19133 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:25.003145 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:25.003393 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:25.006237 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "e840e60c19134f71a5ba3ffdc20a24f4"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "62e3211147038fce861e8846109a02ca"
server_key_iv: "babfdecf42040151a7148b2b38ea2319"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.006973 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "e840e60c19134f71a5ba3ffdc20a24f4"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "62e3211147038fce861e8846109a02ca"
server_key_iv: "babfdecf42040151a7148b2b38ea2319"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.011934 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:34:25.015162 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.016508 19133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:25.016650 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "e840e60c19134f71a5ba3ffdc20a24f4"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "62e3211147038fce861e8846109a02ca"
server_key_iv: "babfdecf42040151a7148b2b38ea2319"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.016817 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:25.037240 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:25.038048 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:25.038306 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:25.057528 19133 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41251
I20260620 18:34:25.057543 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41251 every 8 connection(s)
I20260620 18:34:25.058876 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:25.062580 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:25.068866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19133
I20260620 18:34:25.069065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:25.069656 388 external_mini_cluster.cc:1428] Setting key 48c90b3b6d29a5e4ac34a26c3ab028e0
I20260620 18:34:25.073614 19199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: Bootstrap starting.
I20260620 18:34:25.076752 19199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:25.077724 19199 log.cc:824] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:25.080991 19199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: No bootstrap required, opened a new log
I20260620 18:34:25.083899 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:25.071659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.97.62:41251)
0620 18:34:25.072603 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:25.072615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:25.072654 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:34:25.074415 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:25.074437 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:25.074892 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:25.075285 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:25.075868 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.077174 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.078791 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.079881 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.080074 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:25.081135 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:25.081145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:25.081162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:25.081893 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:25.081985 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:25.082015 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:25.082126 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:25.082309 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:34:25.082622 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":312,"threads_started":1}
I20260620 18:34:25.089928 19199 raft_consensus.cc:348] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } }
I20260620 18:34:25.090238 19199 raft_consensus.cc:374] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:25.090312 19199 raft_consensus.cc:729] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e840e60c19134f71a5ba3ffdc20a24f4, State: Initialized, Role: FOLLOWER
I20260620 18:34:25.090833 19199 consensus_queue.cc:260] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } }
I20260620 18:34:25.091037 19199 raft_consensus.cc:388] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:25.091131 19199 raft_consensus.cc:482] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:25.091250 19199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:25.093302 19199 raft_consensus.cc:504] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } }
I20260620 18:34:25.093717 19199 leader_election.cc:302] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e840e60c19134f71a5ba3ffdc20a24f4; no voters:
I20260620 18:34:25.094264 19199 leader_election.cc:288] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:25.094439 19204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:25.095556 19204 raft_consensus.cc:686] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [term 1 LEADER]: Becoming Leader. State: Replica: e840e60c19134f71a5ba3ffdc20a24f4, State: Running, Role: LEADER
I20260620 18:34:25.096024 19204 consensus_queue.cc:237] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } }
I20260620 18:34:25.096607 19199 sys_catalog.cc:564] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:25.098837 19206 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e840e60c19134f71a5ba3ffdc20a24f4. Latest consensus state: current_term: 1 leader_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } } }
I20260620 18:34:25.099344 19206 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:25.099731 19205 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840e60c19134f71a5ba3ffdc20a24f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41251 } } }
I20260620 18:34:25.100090 19205 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:25.105026 19213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:25.110411 19213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:25.118693 19213 catalog_manager.cc:1347] Generated new cluster ID: 30d7d349194c4b57bb59d1a1262278e0
I20260620 18:34:25.118813 19213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:25.131914 19213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:25.133131 19213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:25.141656 19213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: Generated new TSK 0
I20260620 18:34:25.142675 19213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:25.156840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--builtin_ntp_servers=127.0.97.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:25.483561 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:25.483819 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:25.483903 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:25.490983 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:25.491096 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:25.491200 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:25.499890 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:25.501034 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:25.502691 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:25.516039 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.516010 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.516362 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:25.517202 19223 server_base.cc:1034] running on GCE node
I20260620 18:34:25.517854 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:25.518801 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:25.520040 19223 hybrid_clock.cc:648] HybridClock initialized: now 1781980465519992 us; error 35 us; skew 500 ppm
I20260620 18:34:25.520427 19223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:25.522837 19223 webserver.cc:460] Webserver started at http://127.0.97.1:41401/ using document root <none> and password file <none>
I20260620 18:34:25.523660 19223 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:25.523788 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:25.524156 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:25.527174 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "6ae22955121d4687a06c5584a47f5248"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "6705e7207a34bb3bee5c6908fc26795c"
server_key_iv: "b799276c63b03c8f98a174ad8d8faebc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.528043 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6ae22955121d4687a06c5584a47f5248"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "6705e7207a34bb3bee5c6908fc26795c"
server_key_iv: "b799276c63b03c8f98a174ad8d8faebc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.533285 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:25.536720 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.538010 19223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:34:25.538215 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "6ae22955121d4687a06c5584a47f5248"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "6705e7207a34bb3bee5c6908fc26795c"
server_key_iv: "b799276c63b03c8f98a174ad8d8faebc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.538450 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:25.555131 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:25.556178 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:25.556499 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:25.557528 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:25.559016 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:25.559104 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.559186 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:25.559242 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.593004 19223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44145
I20260620 18:34:25.593080 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44145 every 8 connection(s)
I20260620 18:34:25.594347 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:25.599833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19223
I20260620 18:34:25.599993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:25.600538 388 external_mini_cluster.cc:1428] Setting key 4d2fcd0a501e9111c4764322d60c5376
I20260620 18:34:25.603179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--builtin_ntp_servers=127.0.97.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:25.608946 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:25.596942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36013 (local address 127.0.97.62:41251)
0620 18:34:25.597418 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:25.597425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:25.598774 (+ 1349us) server_negotiation.cc:407] Connection header received
0620 18:34:25.601025 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:25.601031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:25.601136 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:25.601268 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:25.602742 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.603707 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.605844 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.606302 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.606451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:25.607992 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:25.607998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:25.608002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:25.608022 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:25.608064 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:25.608077 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:25.608184 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:25.608435 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:34:25.608541 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":218,"threads_started":1}
I20260620 18:34:25.610060 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:25.597604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41251 (local address 127.0.97.1:36013)
0620 18:34:25.598617 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:25.598655 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:25.600731 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:25.601731 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:25.601741 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:25.601837 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:25.602567 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:25.602585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.603864 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.603872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:25.605689 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:25.605702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.606547 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.606557 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:25.606817 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:25.607632 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:25.607634 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:25.607677 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:25.607762 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:25.608182 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:25.608195 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:25.608207 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:34:25.608426 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 18:34:25.608859 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":347,"threads_started":1}
I20260620 18:34:25.611943 19350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41251
I20260620 18:34:25.612279 19350 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:25.613126 19350 heartbeater.cc:511] Master 127.0.97.62:41251 requested a full tablet report, sending...
I20260620 18:34:25.615456 19164 ts_manager.cc:194] Registered new tserver with Master: 6ae22955121d4687a06c5584a47f5248 (127.0.97.1:44145)
I20260620 18:34:25.617705 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36013
W20260620 18:34:25.928194 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:25.928476 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:25.928570 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:25.935634 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:25.935739 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:25.935842 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:25.945122 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:25.946337 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:25.948081 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:25.961409 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.961385 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.961506 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:25.962260 19354 server_base.cc:1034] running on GCE node
I20260620 18:34:25.962860 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:25.963892 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:25.965129 19354 hybrid_clock.cc:648] HybridClock initialized: now 1781980465965086 us; error 56 us; skew 500 ppm
I20260620 18:34:25.965461 19354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:25.967933 19354 webserver.cc:460] Webserver started at http://127.0.97.2:34015/ using document root <none> and password file <none>
I20260620 18:34:25.968642 19354 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:25.968783 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:25.969151 19354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:25.972110 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "648478026d114b9fb7ed57aae4aad1cf"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "4a822a3c74680fa55560b910aef3098d"
server_key_iv: "74126eb3171bd8a3597073cd9b1ee2b6"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.972878 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "648478026d114b9fb7ed57aae4aad1cf"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "4a822a3c74680fa55560b910aef3098d"
server_key_iv: "74126eb3171bd8a3597073cd9b1ee2b6"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.977861 19354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:34:25.981288 19368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.982586 19354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:34:25.982760 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "648478026d114b9fb7ed57aae4aad1cf"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-mcnm"
server_key: "4a822a3c74680fa55560b910aef3098d"
server_key_iv: "74126eb3171bd8a3597073cd9b1ee2b6"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.982976 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:25.996045 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:25.996946 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:25.997220 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:25.998212 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:25.999698 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:25.999818 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.999953 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:26.000048 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.033449 19354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37467
I20260620 18:34:26.033499 19480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37467 every 8 connection(s)
I20260620 18:34:26.034758 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:26.038846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19354
I20260620 18:34:26.039032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:26.039656 388 external_mini_cluster.cc:1428] Setting key 60a800165e42258f7f4a933a84d923a7
I20260620 18:34:26.042840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--builtin_ntp_servers=127.0.97.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:26.049921 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.037410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41889 (local address 127.0.97.62:41251)
0620 18:34:26.037602 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.037607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.039300 (+ 1693us) server_negotiation.cc:407] Connection header received
0620 18:34:26.041499 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.041504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.041586 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.041693 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.043122 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.044332 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.046879 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.047325 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.047432 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.049112 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.049117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.049121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:26.049154 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.049187 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:26.049199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.049273 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.049598 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.049681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:26.051299 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.037985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41251 (local address 127.0.97.2:41889)
0620 18:34:26.039121 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.039177 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.041189 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.041979 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.041990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.042108 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:26.042906 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.042935 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.044549 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.044557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.046700 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.046717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.047579 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.047589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.047910 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.048746 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:26.048752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:26.048804 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:26.048906 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:26.049299 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:26.049312 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:26.049323 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.049505 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.049920 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":304,"threads_started":1}
I20260620 18:34:26.052798 19481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41251
I20260620 18:34:26.053140 19481 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:26.053895 19481 heartbeater.cc:511] Master 127.0.97.62:41251 requested a full tablet report, sending...
I20260620 18:34:26.055397 19164 ts_manager.cc:194] Registered new tserver with Master: 648478026d114b9fb7ed57aae4aad1cf (127.0.97.2:37467)
I20260620 18:34:26.056883 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41889
W20260620 18:34:26.372434 19484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:26.372730 19484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:26.372877 19484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:26.380174 19484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:26.380268 19484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:26.380424 19484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:26.390141 19484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:26.391409 19484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:26.393184 19484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:26.405944 19489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.406123 19490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.406454 19492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:26.407440 19484 server_base.cc:1034] running on GCE node
I20260620 18:34:26.408103 19484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:26.409116 19484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:26.410403 19484 hybrid_clock.cc:648] HybridClock initialized: now 1781980466410321 us; error 79 us; skew 500 ppm
I20260620 18:34:26.410789 19484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:26.413246 19484 webserver.cc:460] Webserver started at http://127.0.97.3:44729/ using document root <none> and password file <none>
I20260620 18:34:26.413967 19484 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:26.414106 19484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:26.414431 19484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:26.417402 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "0c717988d3cb40e2ab3e92e916dafd24"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-mcnm"
server_key: "2d599c895ba02f87b20bfb2604c92b78"
server_key_iv: "dd1a587b42d410865a63dc9a4222b98b"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.418216 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0c717988d3cb40e2ab3e92e916dafd24"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-mcnm"
server_key: "2d599c895ba02f87b20bfb2604c92b78"
server_key_iv: "dd1a587b42d410865a63dc9a4222b98b"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.423492 19484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:34:26.426970 19498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.428350 19484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:34:26.428529 19484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "0c717988d3cb40e2ab3e92e916dafd24"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-mcnm"
server_key: "2d599c895ba02f87b20bfb2604c92b78"
server_key_iv: "dd1a587b42d410865a63dc9a4222b98b"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.428758 19484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:26.450627 19484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:26.451457 19484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:26.451766 19484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:26.452785 19484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:26.454294 19484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:26.454444 19484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.454577 19484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:26.454665 19484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.488054 19484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40093
I20260620 18:34:26.488142 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40093 every 8 connection(s)
I20260620 18:34:26.489357 19484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:26.491026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19484
I20260620 18:34:26.491219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:26.491740 388 external_mini_cluster.cc:1428] Setting key 0773b6a3718a05ad9821d10c2ee30152
I20260620 18:34:26.503641 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.492338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39835 (local address 127.0.97.62:41251)
0620 18:34:26.492561 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.492568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.494471 (+ 1903us) server_negotiation.cc:407] Connection header received
0620 18:34:26.496583 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.496589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.496707 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.496842 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.498013 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.498849 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.500697 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.501184 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.501360 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.502808 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.502813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.502816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:26.502833 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.502863 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:26.502874 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.502978 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.503194 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.503294 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:34:26.504814 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.493046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41251 (local address 127.0.97.3:39835)
0620 18:34:26.494294 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.494345 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.496289 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.497105 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.497114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.497216 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:26.497817 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.497835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.499056 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.499061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.500523 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.500538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.501450 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.501458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.501728 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.502447 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:26.502450 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:26.502487 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:26.502565 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:26.502976 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:26.502987 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:26.502998 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.503180 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.503623 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":461,"threads_started":1}
I20260620 18:34:26.506255 19611 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41251
I20260620 18:34:26.506613 19611 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:26.507588 19611 heartbeater.cc:511] Master 127.0.97.62:41251 requested a full tablet report, sending...
I20260620 18:34:26.509014 19164 ts_manager.cc:194] Registered new tserver with Master: 0c717988d3cb40e2ab3e92e916dafd24 (127.0.97.3:40093)
I20260620 18:34:26.509774 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39835
I20260620 18:34:26.519193 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:26.529029 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.523431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.97.62:41251)
0620 18:34:26.523664 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.523669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.524152 (+ 483us) server_negotiation.cc:407] Connection header received
0620 18:34:26.524316 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.524320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.524397 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.524494 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.524895 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.525567 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.526833 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.527202 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.527294 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.528386 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.528390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.528394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:26.528411 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.528441 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:26.528453 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.528583 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.528760 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.528804 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:34:26.532874 19164 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:26.539690 19164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:26.569096 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.551766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54588)
0620 18:34:26.552920 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.552977 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.553200 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.555506 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.555538 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.555590 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:26.556139 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.556154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.559128 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.559137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.564921 (+ 5784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.564939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.567687 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.567701 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.568213 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.568264 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.568361 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.568551 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":2801,"thread_start_us":3954,"threads_started":1}
I20260620 18:34:26.569759 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.552826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.97.3:40093)
0620 18:34:26.553994 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.554016 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.556317 (+ 2301us) server_negotiation.cc:407] Connection header received
0620 18:34:26.556418 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.556426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.556652 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.556885 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:26.557910 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.559990 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.563313 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.566806 (+ 3493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.566973 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.567361 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.568684 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.568924 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":1268,"threads_started":1}
I20260620 18:34:26.570219 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.551800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.0.97.1:44145)
0620 18:34:26.554976 (+ 3176us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.554983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.555008 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:26.555091 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.555096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.555282 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.555428 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:26.556345 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.558926 (+ 2581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.565203 (+ 6277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.567190 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.567359 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.568006 (+ 647us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.569746 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.569915 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3034,"thread_start_us":2934,"threads_started":1}
I20260620 18:34:26.571357 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.552107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40093 (local address 127.0.0.1:57800)
0620 18:34:26.556033 (+ 3926us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.556056 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.556195 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.557227 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.557232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.557255 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:26.557676 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.557691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.560195 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.560203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.563083 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.563105 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.567534 (+ 4429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.567543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.567912 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.567986 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.568143 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.568395 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3756,"mutex_wait_us":1,"thread_start_us":1035,"threads_started":1}
I20260620 18:34:26.574426 19285 tablet_service.cc:1467] Processing CreateTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d (DEFAULT_TABLE table=test-table [id=9dd7cc6f3d474c02a6e509aba0052234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:26.574913 19546 tablet_service.cc:1467] Processing CreateTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d (DEFAULT_TABLE table=test-table [id=9dd7cc6f3d474c02a6e509aba0052234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:26.575910 19285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42f5e979a6b4a3f8d58f3c5214def1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:26.576195 19546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42f5e979a6b4a3f8d58f3c5214def1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:26.577499 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.556293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.0.1:52206)
0620 18:34:26.569408 (+ 13115us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.569432 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.569549 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.570802 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.570809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.570839 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:26.571062 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.571074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.572892 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.572897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.574647 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.574674 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.576698 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.576704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.576937 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.576980 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.577085 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.577231 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12984,"thread_start_us":14461,"threads_started":1}
I20260620 18:34:26.578249 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.556524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.0.97.2:37467)
0620 18:34:26.569922 (+ 13398us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.569930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.569958 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:34:26.570053 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.570063 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.570344 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.570546 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:26.571239 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.572735 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.574850 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.576248 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.576385 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.576601 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.577732 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.577914 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13233,"thread_start_us":13921,"threads_started":1}
I20260620 18:34:26.581969 19416 tablet_service.cc:1467] Processing CreateTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d (DEFAULT_TABLE table=test-table [id=9dd7cc6f3d474c02a6e509aba0052234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:26.583315 19416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42f5e979a6b4a3f8d58f3c5214def1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:26.587622 19629 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: Bootstrap starting.
I20260620 18:34:26.590497 19629 tablet_bootstrap.cc:654] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:26.591470 19630 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Bootstrap starting.
I20260620 18:34:26.591889 19629 log.cc:824] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:26.596136 19629 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: No bootstrap required, opened a new log
I20260620 18:34:26.596495 19629 ts_tablet_manager.cc:1397] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:34:26.596606 19630 tablet_bootstrap.cc:654] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:26.597759 19630 log.cc:824] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:26.600641 19633 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Bootstrap starting.
I20260620 18:34:26.600955 19630 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: No bootstrap required, opened a new log
I20260620 18:34:26.601243 19630 ts_tablet_manager.cc:1397] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:34:26.604128 19633 tablet_bootstrap.cc:654] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:26.605374 19633 log.cc:824] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:26.609072 19629 raft_consensus.cc:348] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.609647 19629 raft_consensus.cc:374] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:26.609817 19629 raft_consensus.cc:729] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae22955121d4687a06c5584a47f5248, State: Initialized, Role: FOLLOWER
I20260620 18:34:26.610608 19629 consensus_queue.cc:260] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.611809 19350 heartbeater.cc:503] Master 127.0.97.62:41251 was elected leader, sending a full tablet report...
I20260620 18:34:26.612433 19633 tablet_bootstrap.cc:492] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: No bootstrap required, opened a new log
I20260620 18:34:26.612483 19629 ts_tablet_manager.cc:1428] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260620 18:34:26.612694 19633 ts_tablet_manager.cc:1397] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 18:34:26.612962 19630 raft_consensus.cc:348] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.613411 19630 raft_consensus.cc:374] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:26.613511 19630 raft_consensus.cc:729] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c717988d3cb40e2ab3e92e916dafd24, State: Initialized, Role: FOLLOWER
I20260620 18:34:26.614094 19630 consensus_queue.cc:260] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.615765 19611 heartbeater.cc:503] Master 127.0.97.62:41251 was elected leader, sending a full tablet report...
I20260620 18:34:26.615826 19630 ts_tablet_manager.cc:1428] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260620 18:34:26.622301 19633 raft_consensus.cc:348] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.622653 19633 raft_consensus.cc:374] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:26.622778 19633 raft_consensus.cc:729] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648478026d114b9fb7ed57aae4aad1cf, State: Initialized, Role: FOLLOWER
I20260620 18:34:26.623329 19636 raft_consensus.cc:482] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:26.623373 19633 consensus_queue.cc:260] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.623624 19636 raft_consensus.cc:504] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.624533 19481 heartbeater.cc:503] Master 127.0.97.62:41251 was elected leader, sending a full tablet report...
I20260620 18:34:26.625182 19633 ts_tablet_manager.cc:1428] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 18:34:26.625550 19636 leader_election.cc:288] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae22955121d4687a06c5584a47f5248 (127.0.97.1:44145), 648478026d114b9fb7ed57aae4aad1cf (127.0.97.2:37467)
I20260620 18:34:26.632856 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.625953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.97.3:39847)
0620 18:34:26.626129 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.626150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.626261 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.626825 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.626833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.626863 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:26.627083 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.627098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.628513 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.628518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.630621 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.630635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.632141 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.632153 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.632283 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.632345 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.632432 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.632562 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":25}
I20260620 18:34:26.632859 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.625989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39847 (local address 127.0.97.1:44145)
0620 18:34:26.626196 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.626204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.626228 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:34:26.626449 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.626456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.626538 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.626637 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:26.627263 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.628365 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.630770 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.631855 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.632147 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.632403 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.632478 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.632607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:26.633987 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.626987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34245 (local address 127.0.97.2:37467)
0620 18:34:26.627210 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.627217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.627240 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:34:26.627374 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.627380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.627478 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.627603 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:26.628171 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.629199 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.631374 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.632911 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.633209 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.633297 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.633621 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.633752 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:34:26.634032 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.626374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.97.3:34245)
0620 18:34:26.627084 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.627105 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.627257 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.627620 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.627625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.627651 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:26.627979 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.627989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.629421 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.629426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.631196 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.631205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.633264 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.633270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.633455 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.633502 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.633584 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.633702 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":3,"thread_start_us":344,"threads_started":1}
I20260620 18:34:26.634230 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42f5e979a6b4a3f8d58f3c5214def1d" candidate_uuid: "0c717988d3cb40e2ab3e92e916dafd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae22955121d4687a06c5584a47f5248" is_pre_election: true
I20260620 18:34:26.634706 19305 raft_consensus.cc:2454] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c717988d3cb40e2ab3e92e916dafd24 in term 0.
I20260620 18:34:26.635140 19436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42f5e979a6b4a3f8d58f3c5214def1d" candidate_uuid: "0c717988d3cb40e2ab3e92e916dafd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648478026d114b9fb7ed57aae4aad1cf" is_pre_election: true
I20260620 18:34:26.635310 19499 leader_election.cc:302] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c717988d3cb40e2ab3e92e916dafd24, 6ae22955121d4687a06c5584a47f5248; no voters:
I20260620 18:34:26.635653 19436 raft_consensus.cc:2454] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c717988d3cb40e2ab3e92e916dafd24 in term 0.
I20260620 18:34:26.635727 19636 raft_consensus.cc:2781] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:26.635854 19636 raft_consensus.cc:482] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:26.635969 19636 raft_consensus.cc:3037] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:26.638106 19636 raft_consensus.cc:504] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.638692 19636 leader_election.cc:288] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae22955121d4687a06c5584a47f5248 (127.0.97.1:44145), 648478026d114b9fb7ed57aae4aad1cf (127.0.97.2:37467)
I20260620 18:34:26.639122 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42f5e979a6b4a3f8d58f3c5214def1d" candidate_uuid: "0c717988d3cb40e2ab3e92e916dafd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae22955121d4687a06c5584a47f5248"
I20260620 18:34:26.639207 19436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42f5e979a6b4a3f8d58f3c5214def1d" candidate_uuid: "0c717988d3cb40e2ab3e92e916dafd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648478026d114b9fb7ed57aae4aad1cf"
I20260620 18:34:26.639321 19305 raft_consensus.cc:3037] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:26.639408 19436 raft_consensus.cc:3037] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:26.641494 19305 raft_consensus.cc:2454] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c717988d3cb40e2ab3e92e916dafd24 in term 1.
I20260620 18:34:26.641768 19436 raft_consensus.cc:2454] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c717988d3cb40e2ab3e92e916dafd24 in term 1.
I20260620 18:34:26.641847 19499 leader_election.cc:302] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c717988d3cb40e2ab3e92e916dafd24, 6ae22955121d4687a06c5584a47f5248; no voters:
I20260620 18:34:26.642158 19636 raft_consensus.cc:2781] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:26.642652 19636 raft_consensus.cc:686] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 1 LEADER]: Becoming Leader. State: Replica: 0c717988d3cb40e2ab3e92e916dafd24, State: Running, Role: LEADER
I20260620 18:34:26.643206 19636 consensus_queue.cc:237] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } }
I20260620 18:34:26.647780 19164 catalog_manager.cc:5495] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c717988d3cb40e2ab3e92e916dafd24 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0c717988d3cb40e2ab3e92e916dafd24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c717988d3cb40e2ab3e92e916dafd24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40093 } health_report { overall_health: HEALTHY } } }
I20260620 18:34:26.664259 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.658972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.0.97.3:40093)
0620 18:34:26.659144 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.659149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.659165 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:34:26.659297 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.659301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.659382 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.659499 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:26.659867 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.660563 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.662105 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.662508 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.662594 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.663080 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:26.663613 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:26.663743 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.663914 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.663975 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:26.672039 19436 raft_consensus.cc:1261] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 1 FOLLOWER]: Refusing update from remote peer 0c717988d3cb40e2ab3e92e916dafd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:26.672051 19305 raft_consensus.cc:1261] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248 [term 1 FOLLOWER]: Refusing update from remote peer 0c717988d3cb40e2ab3e92e916dafd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:26.672951 19639 consensus_queue.cc:1035] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae22955121d4687a06c5584a47f5248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:26.673323 19636 consensus_queue.cc:1035] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648478026d114b9fb7ed57aae4aad1cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:26.704157 19163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:26.704514 19163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:26.708535 19163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e840e60c19134f71a5ba3ffdc20a24f4: Sending DeleteTablet for 3 replicas of tablet d42f5e979a6b4a3f8d58f3c5214def1d
I20260620 18:34:26.709826 19285 tablet_service.cc:1514] Processing DeleteTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:26 UTC) from {username='slave'} at 127.0.0.1:54588
I20260620 18:34:26.709954 19416 tablet_service.cc:1514] Processing DeleteTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:26 UTC) from {username='slave'} at 127.0.0.1:52206
I20260620 18:34:26.710717 19546 tablet_service.cc:1514] Processing DeleteTablet for tablet d42f5e979a6b4a3f8d58f3c5214def1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:26 UTC) from {username='slave'} at 127.0.0.1:57800
I20260620 18:34:26.711277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19223
I20260620 18:34:26.713830 19654 tablet_replica.cc:330] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: stopping tablet replica
I20260620 18:34:26.714310 19654 raft_consensus.cc:2229] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:26.715003 19654 raft_consensus.cc:2258] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:26.721485 19654 ts_tablet_manager.cc:1905] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:26.722993 19653 tablet_replica.cc:330] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: stopping tablet replica
I20260620 18:34:26.723625 19653 raft_consensus.cc:2229] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:26.724321 19653 raft_consensus.cc:2258] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:26.725493 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.713725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.97.1:44145)
0620 18:34:26.713926 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.713935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.716058 (+ 2123us) server_negotiation.cc:407] Connection header received
0620 18:34:26.716342 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.716347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.716452 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.716528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.717189 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.718451 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.720460 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.721101 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.721202 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.723598 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.723610 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.723628 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:26.724429 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.724497 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:26.724522 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.724605 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.724935 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.725051 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:26.726904 19345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:26.727207 19653 ts_tablet_manager.cc:1905] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:26.728866 19654 ts_tablet_manager.cc:1918] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:26.729041 19654 log.cc:1196] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/d42f5e979a6b4a3f8d58f3c5214def1d
I20260620 18:34:26.729606 19654 ts_tablet_manager.cc:1939] T d42f5e979a6b4a3f8d58f3c5214def1d P 0c717988d3cb40e2ab3e92e916dafd24: Deleting consensus metadata
I20260620 18:34:26.731420 19150 catalog_manager.cc:4841] TS 0c717988d3cb40e2ab3e92e916dafd24 (127.0.97.3:40093): tablet d42f5e979a6b4a3f8d58f3c5214def1d (table test-table [id=9dd7cc6f3d474c02a6e509aba0052234]) successfully deleted
I20260620 18:34:26.737113 19653 ts_tablet_manager.cc:1918] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:26.737322 19653 log.cc:1196] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/d42f5e979a6b4a3f8d58f3c5214def1d
I20260620 18:34:26.737978 19653 ts_tablet_manager.cc:1939] T d42f5e979a6b4a3f8d58f3c5214def1d P 648478026d114b9fb7ed57aae4aad1cf: Deleting consensus metadata
I20260620 18:34:26.740367 19148 catalog_manager.cc:4841] TS 648478026d114b9fb7ed57aae4aad1cf (127.0.97.2:37467): tablet d42f5e979a6b4a3f8d58f3c5214def1d (table test-table [id=9dd7cc6f3d474c02a6e509aba0052234]) successfully deleted
I20260620 18:34:26.877965 19652 tablet_replica.cc:330] T d42f5e979a6b4a3f8d58f3c5214def1d P 6ae22955121d4687a06c5584a47f5248: stopping tablet replica
I20260620 18:34:26.878106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19223
W20260620 18:34:26.896014 19148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44145 (error 108)
I20260620 18:34:26.896755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19354
I20260620 18:34:26.898242 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.897621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54608)
0620 18:34:26.897878 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.898005 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 18:34:26.898540 19148 catalog_manager.cc:4584] TS 6ae22955121d4687a06c5584a47f5248 (127.0.97.1:44145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d42f5e979a6b4a3f8d58f3c5214def1d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
I20260620 18:34:26.905102 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.897641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.97.2:37467)
0620 18:34:26.897913 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.897919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.897991 (+ 72us) server_negotiation.cc:407] Connection header received
0620 18:34:26.898227 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.898235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.898355 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.898462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.898962 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.899759 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.901861 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.902244 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.902338 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.903480 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.903488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.903502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:26.904336 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.904384 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:26.904403 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.904487 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.904651 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.904759 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 18:34:26.905843 19476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:26.952581 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.952009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54612)
0620 18:34:26.952271 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.952389 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:34:26.953807 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.953370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54618)
0620 18:34:26.953544 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.953631 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:34:26.996251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19354
I20260620 18:34:27.013881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19484
I20260620 18:34:27.014544 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.013936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54626)
0620 18:34:27.014233 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:27.014336 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 18:34:27.015827 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.015279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54642)
0620 18:34:27.015506 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:27.015644 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 18:34:27.021687 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.014721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.0.97.3:40093)
0620 18:34:27.014905 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:27.014912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:27.015176 (+ 264us) server_negotiation.cc:407] Connection header received
0620 18:34:27.015388 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:27.015393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:27.015496 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:27.015651 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:27.016134 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.016939 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.018274 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.018729 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.018878 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:27.020184 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:27.020192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:27.020205 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:27.020893 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:27.020939 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:27.020958 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:27.021100 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:27.021212 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:34:27.021310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:34:27.022449 19606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:27.096460 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.096018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54648)
0620 18:34:27.096228 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:27.096314 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:34:27.097692 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.097201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44145 (local address 127.0.0.1:54660)
0620 18:34:27.097443 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:27.097564 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 18:34:27.142845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19484
I20260620 18:34:27.162345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19133
I20260620 18:34:27.163565 19194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:27.273425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19133
2026-06-20T18:34:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2723 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:27 dist-test-slave-mcnm krb5kdc[19663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:27 dist-test-slave-mcnm krb5kdc[19663](info): set up 2 sockets
Jun 20 18:34:27 dist-test-slave-mcnm krb5kdc[19663](info): commencing operation
krb5kdc: starting...
W20260620 18:34:28.395759 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:28 dist-test-slave-mcnm krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:34:28.643056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42685
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:28.975907 19676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:28.976229 19676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:28.976380 19676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:28.983913 19676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:28.984031 19676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:28.984081 19676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:28.984133 19676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:28.984210 19676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:28.994270 19676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:28.995671 19676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:28.997015 19676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.009279 19681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.009271 19684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.009312 19682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.010401 19676 server_base.cc:1034] running on GCE node
I20260620 18:34:29.011178 19676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.012687 19676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.013911 19676 hybrid_clock.cc:648] HybridClock initialized: now 1781980469013870 us; error 31 us; skew 500 ppm
I20260620 18:34:29.014307 19676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:29 dist-test-slave-mcnm krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.023192 19676 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:34:29.024951 19676 webserver.cc:460] Webserver started at http://127.0.97.62:42763/ using document root <none> and password file <none>
I20260620 18:34:29.025703 19676 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.025857 19676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.026178 19676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.029183 19676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "8f7d541a03d74d13aea1cce64edc16ab"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "14e66e561f8dcc4d29ea78ddef8a7d21"
server_key_iv: "4139279eba133216a69f7e712e40152f"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.029943 19676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f7d541a03d74d13aea1cce64edc16ab"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "14e66e561f8dcc4d29ea78ddef8a7d21"
server_key_iv: "4139279eba133216a69f7e712e40152f"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.035023 19676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:34:29.038393 19691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.039875 19676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:34:29.040063 19676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "8f7d541a03d74d13aea1cce64edc16ab"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "14e66e561f8dcc4d29ea78ddef8a7d21"
server_key_iv: "4139279eba133216a69f7e712e40152f"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.040298 19676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.054059 19676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.058616 19676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.058909 19676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.077600 19676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42685
I20260620 18:34:29.077638 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42685 every 8 connection(s)
I20260620 18:34:29.079010 19676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:29.082600 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:29.086295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19676
I20260620 18:34:29.086480 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:29.087069 388 external_mini_cluster.cc:1428] Setting key 3ecc447c35a7e66703c052f7c5a0570b
I20260620 18:34:29.095603 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: Bootstrap starting.
I20260620 18:34:29.099337 19743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:29.100759 19743 log.cc:824] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:34:29 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:29.103705 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: No bootstrap required, opened a new log
I20260620 18:34:29.110667 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.089008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.97.62:42685)
0620 18:34:29.089797 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.089808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.089848 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:34:29.091481 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.091502 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.091951 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.092270 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.094054 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.095359 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.097255 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.097968 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.098101 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.103266 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.103273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.103286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.103323 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.106506 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.107255 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.107262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.107268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.107450 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.107889 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.107894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.107898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.108431 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.108763 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.109252 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.109679 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":380,"threads_started":1}
I20260620 18:34:29.111466 19743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } }
I20260620 18:34:29.111931 19743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:29.112057 19743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7d541a03d74d13aea1cce64edc16ab, State: Initialized, Role: FOLLOWER
I20260620 18:34:29.112735 19743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } }
I20260620 18:34:29.113047 19743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:29.113191 19743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:29.113371 19743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:29.115784 19743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } }
I20260620 18:34:29.116271 19743 leader_election.cc:302] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f7d541a03d74d13aea1cce64edc16ab; no voters:
I20260620 18:34:29.116850 19743 leader_election.cc:288] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:29.117031 19748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:29.117740 19748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [term 1 LEADER]: Becoming Leader. State: Replica: 8f7d541a03d74d13aea1cce64edc16ab, State: Running, Role: LEADER
I20260620 18:34:29.118299 19748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } }
I20260620 18:34:29.119197 19743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:29.121325 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7d541a03d74d13aea1cce64edc16ab. Latest consensus state: current_term: 1 leader_uuid: "8f7d541a03d74d13aea1cce64edc16ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } } }
I20260620 18:34:29.121866 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:29.121356 19749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f7d541a03d74d13aea1cce64edc16ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f7d541a03d74d13aea1cce64edc16ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42685 } } }
I20260620 18:34:29.122406 19749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:29.127950 19755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:29.132726 19755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:29.140997 19755 catalog_manager.cc:1347] Generated new cluster ID: 3c5ff210209c4be7b0fdd5334b10b67a
I20260620 18:34:29.141110 19755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:29.178231 19755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:29.180141 19755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:29.188987 19755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: Generated new TSK 0
I20260620 18:34:29.189884 19755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 18:34:29.263177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:29.593708 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:29.593974 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:29.594138 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:29.601398 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:29.601500 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:29.601673 19769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:29.611258 19769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:29.612548 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:29.613904 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.627437 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.627662 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.627832 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.628855 19769 server_base.cc:1034] running on GCE node
I20260620 18:34:29.629447 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.630393 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.631620 19769 hybrid_clock.cc:648] HybridClock initialized: now 1781980469631580 us; error 32 us; skew 500 ppm
I20260620 18:34:29.631961 19769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:29 dist-test-slave-mcnm krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.635102 19769 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 18:34:29.636688 19769 webserver.cc:460] Webserver started at http://127.0.97.1:36351/ using document root <none> and password file <none>
I20260620 18:34:29.637374 19769 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.637501 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.637823 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.640767 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b8b57ba2b754b4abc90b8c56d643208"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "bcea8750658161768439923e2bd16e17"
server_key_iv: "7c98f3ac8a464a0d585e3e5cbc7a5e11"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.641563 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b8b57ba2b754b4abc90b8c56d643208"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "bcea8750658161768439923e2bd16e17"
server_key_iv: "7c98f3ac8a464a0d585e3e5cbc7a5e11"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.646824 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:34:29.650352 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.651844 19769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:29.652014 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "3b8b57ba2b754b4abc90b8c56d643208"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-mcnm"
server_key: "bcea8750658161768439923e2bd16e17"
server_key_iv: "7c98f3ac8a464a0d585e3e5cbc7a5e11"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.652220 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.691020 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.695447 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.695780 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.696784 19769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:29.698129 19769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:29.698259 19769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.698398 19769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:29.698472 19769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.733614 19769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33915
I20260620 18:34:29.733669 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33915 every 8 connection(s)
I20260620 18:34:29.734936 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:29.739895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19769
I20260620 18:34:29.740077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:29.740513 388 external_mini_cluster.cc:1428] Setting key 96c0ad7a4fab4b5cae13b81401fb443d
Jun 20 18:34:29 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:29.756042 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.737658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57035 (local address 127.0.97.62:42685)
0620 18:34:29.738097 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.738101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.739106 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 18:34:29.741232 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.741240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.741342 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.741462 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.742991 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.743808 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.745492 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.745897 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.746042 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.749587 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.749591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.749595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.749628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.752346 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.753297 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.753301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.753303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.753429 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.753981 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.753989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.753991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.754387 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.754537 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.755482 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.755729 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260620 18:34:29.757560 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.738225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42685 (local address 127.0.97.1:57035)
0620 18:34:29.738924 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:29.738963 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:29.740946 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:29.741712 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:29.741721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:29.742232 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:29.742814 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.742836 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.743974 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.743979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.745324 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.745340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.746129 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.746139 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.746391 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.747100 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:29.747123 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:29.749360 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:29.752491 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.752502 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.752523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.753168 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.753545 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.753553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.753560 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.753828 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.754536 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:29.754548 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:29.755015 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 18:34:29.755364 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 18:34:29.756042 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":287,"threads_started":1}
I20260620 18:34:29.759271 19897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42685
I20260620 18:34:29.759709 19897 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:29.760656 19897 heartbeater.cc:511] Master 127.0.97.62:42685 requested a full tablet report, sending...
I20260620 18:34:29.764122 19708 ts_manager.cc:194] Registered new tserver with Master: 3b8b57ba2b754b4abc90b8c56d643208 (127.0.97.1:33915)
I20260620 18:34:29.765698 19708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57035
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 18:34:29.807353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:30.133267 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:30.133572 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:30.133746 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:30.140818 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:30.140973 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:30.141142 19903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:30.150942 19903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:30.152278 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:30.153586 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:30.167021 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:30.167156 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:30.167742 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:30.167976 19903 server_base.cc:1034] running on GCE node
I20260620 18:34:30.168629 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:30.169623 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:30.170878 19903 hybrid_clock.cc:648] HybridClock initialized: now 1781980470170829 us; error 62 us; skew 500 ppm
I20260620 18:34:30.171187 19903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:30 dist-test-slave-mcnm krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:30.174404 19903 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 18:34:30.175954 19903 webserver.cc:460] Webserver started at http://127.0.97.2:40231/ using document root <none> and password file <none>
I20260620 18:34:30.176616 19903 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:30.176764 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:30.177076 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:30.180045 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "a53bd4a483554190a728c64fb4792913"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "75bcfca39362613cd7a566b88a076d56"
server_key_iv: "1ee3a420a6125173ea0bd3ef9dcd2f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.180830 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a53bd4a483554190a728c64fb4792913"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "75bcfca39362613cd7a566b88a076d56"
server_key_iv: "1ee3a420a6125173ea0bd3ef9dcd2f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.185770 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:34:30.189262 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.190591 19903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:34:30.190757 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "a53bd4a483554190a728c64fb4792913"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "75bcfca39362613cd7a566b88a076d56"
server_key_iv: "1ee3a420a6125173ea0bd3ef9dcd2f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.190959 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:30.210041 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:30.215330 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:30.215693 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:30.216689 19903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:30.218016 19903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:30.218129 19903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.218271 19903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:30.218341 19903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.254292 19903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37055
I20260620 18:34:30.254350 20030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37055 every 8 connection(s)
I20260620 18:34:30.255671 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:30.261474 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 19903
I20260620 18:34:30.261682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:30.262233 388 external_mini_cluster.cc:1428] Setting key 5f96d689b9484b16fd8f4c92a02d477c
Jun 20 18:34:30 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:30.277267 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.258586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58579 (local address 127.0.97.62:42685)
0620 18:34:30.259046 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.259051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.259958 (+ 907us) server_negotiation.cc:407] Connection header received
0620 18:34:30.262223 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.262227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.262319 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.262443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:30.264369 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.265070 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.266619 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.267049 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.267179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.271018 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:30.271023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:30.271027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:30.271063 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:30.274047 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.274830 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.274836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.274839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.274992 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.275449 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.275458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.275460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.275783 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:30.275936 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.276638 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.276872 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260620 18:34:30.278275 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.258999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42685 (local address 127.0.97.2:58579)
0620 18:34:30.259794 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.259839 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.261910 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.262719 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.262732 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.263463 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:30.264190 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.264222 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.265233 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.265239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.266465 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.266478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.267294 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.267309 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.267591 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.268337 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:30.268364 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:30.270776 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:30.274233 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:30.274242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:30.274259 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:30.274694 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:30.275119 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:30.275124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:30.275128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:30.275325 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:30.275923 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:30.275931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:30.276268 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.276536 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.277090 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":288,"threads_started":1}
I20260620 18:34:30.279894 20031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42685
I20260620 18:34:30.280212 20031 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:30.280953 20031 heartbeater.cc:511] Master 127.0.97.62:42685 requested a full tablet report, sending...
I20260620 18:34:30.282495 19708 ts_manager.cc:194] Registered new tserver with Master: a53bd4a483554190a728c64fb4792913 (127.0.97.2:37055)
I20260620 18:34:30.283823 19708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58579
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 18:34:30.319280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:30.646145 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:30.646425 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:30.646603 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:30.654047 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:30.654156 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:30.654327 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:30.664175 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:30.665496 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:30.666821 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:30.679888 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:30.680115 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:30.680091 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:30.680775 20037 server_base.cc:1034] running on GCE node
I20260620 18:34:30.681418 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:30.682381 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:30.683640 20037 hybrid_clock.cc:648] HybridClock initialized: now 1781980470683586 us; error 27 us; skew 500 ppm
I20260620 18:34:30.684008 20037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:30 dist-test-slave-mcnm krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:30.687184 20037 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 18:34:30.688745 20037 webserver.cc:460] Webserver started at http://127.0.97.3:33845/ using document root <none> and password file <none>
I20260620 18:34:30.689389 20037 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:30.689486 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:30.689738 20037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:30.692589 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "9aabed634e2e4752b091a1862ca6ac81"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "15f965b2700c8a8e112ee7733fc8a13e"
server_key_iv: "5d481e9ce6cbd5fade4c4fc36656d83b"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.693349 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9aabed634e2e4752b091a1862ca6ac81"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "15f965b2700c8a8e112ee7733fc8a13e"
server_key_iv: "5d481e9ce6cbd5fade4c4fc36656d83b"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.698740 20037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:34:30.702502 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.704003 20037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:30.704185 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "9aabed634e2e4752b091a1862ca6ac81"
format_stamp: "Formatted at 2026-06-20 18:34:30 on dist-test-slave-mcnm"
server_key: "15f965b2700c8a8e112ee7733fc8a13e"
server_key_iv: "5d481e9ce6cbd5fade4c4fc36656d83b"
server_key_version: "encryptionkey@0"
I20260620 18:34:30.704397 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:30.746591 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:30.751012 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:30.751328 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:30.752378 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:30.753785 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:30.753890 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.754035 20037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:30.754106 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:30.769626 19897 heartbeater.cc:503] Master 127.0.97.62:42685 was elected leader, sending a full tablet report...
I20260620 18:34:30.795809 20037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41431
I20260620 18:34:30.795852 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41431 every 8 connection(s)
I20260620 18:34:30.797156 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:30.804329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20037
I20260620 18:34:30.804522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:30.805022 388 external_mini_cluster.cc:1428] Setting key 3fd34f985a26a0a43b04cd5915e28b14
Jun 20 18:34:30 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:30.817678 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.799874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48487 (local address 127.0.97.62:42685)
0620 18:34:30.800367 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.800371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.801446 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 18:34:30.803264 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.803268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.803373 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.803475 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:30.805077 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.806075 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.807896 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.808398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.808493 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.811907 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:30.811912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:30.811915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:30.811946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:30.814461 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.815182 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.815186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.815189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.815328 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.815892 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.815897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.815899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.816256 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:30.816403 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.817148 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.817388 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260620 18:34:30.818619 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.800596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42685 (local address 127.0.97.3:48487)
0620 18:34:30.801275 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.801312 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.802994 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.803742 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.803749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.804243 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:30.804896 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.804914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.806237 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.806242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.807748 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.807765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.808580 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.808585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.808851 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.809493 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:30.809515 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:30.811722 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:30.814583 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:30.814589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:30.814602 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:30.815009 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:30.815478 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:30.815482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:30.815485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:30.815731 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:30.816435 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:30.816441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:30.816748 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.816984 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.817398 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":271,"threads_started":1}
I20260620 18:34:30.820209 20165 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42685
I20260620 18:34:30.820585 20165 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:30.821342 20165 heartbeater.cc:511] Master 127.0.97.62:42685 requested a full tablet report, sending...
I20260620 18:34:30.822713 19708 ts_manager.cc:194] Registered new tserver with Master: 9aabed634e2e4752b091a1862ca6ac81 (127.0.97.3:41431)
I20260620 18:34:30.823618 19708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48487
I20260620 18:34:30.832618 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:30.849545 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.836669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.97.62:42685)
0620 18:34:30.836916 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.836926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.837254 (+ 328us) server_negotiation.cc:407] Connection header received
0620 18:34:30.837417 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.837421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.837494 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.837573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:30.838860 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.839717 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.841080 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.841463 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.841553 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.844089 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:30.844094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:30.844097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:30.844129 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:30.846721 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.847358 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.847362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.847364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.847493 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:30.848093 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:30.848097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:30.848100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:30.848429 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:30.848621 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.848994 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.849174 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:34:30.853518 19708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:30.860891 19708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:30.884210 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.873360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33915 (local address 127.0.0.1:41106)
0620 18:34:30.875349 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.875370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.875486 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.876669 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.876678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.876709 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:30.877121 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.877135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.878986 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.878994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.881443 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.881457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.883144 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.883150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.883343 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.883416 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.883571 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.883759 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"mutex_wait_us":1,"thread_start_us":2371,"threads_started":1}
I20260620 18:34:30.884944 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.873484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.97.1:33915)
0620 18:34:30.875769 (+ 2285us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.875778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.875806 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:34:30.875904 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.875915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.876188 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.876381 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:30.877318 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.878844 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.881605 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.882955 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.883105 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.883339 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.884442 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.884605 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2085,"thread_start_us":1934,"threads_started":1}
I20260620 18:34:30.885455 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.874748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41431 (local address 127.0.0.1:47382)
0620 18:34:30.876775 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.876798 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.876940 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.877584 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.877589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.877620 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:30.878189 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.878200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.880220 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.880225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.882354 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.882366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.884682 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.884688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.884931 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.884974 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.885072 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.885199 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1875,"mutex_wait_us":150,"thread_start_us":336,"threads_started":1}
I20260620 18:34:30.886128 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.872724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37055 (local address 127.0.0.1:52892)
0620 18:34:30.874291 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.874327 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.874485 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.877653 (+ 3168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.877660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.877685 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:30.878295 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.878308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.880222 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.880226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.882485 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.882500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.884585 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.884593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.884817 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.884852 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.884904 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.884988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"mutex_wait_us":96,"thread_start_us":1219,"threads_started":1}
I20260620 18:34:30.886194 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.875106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.0.97.3:41431)
0620 18:34:30.876383 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.876400 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.877068 (+ 668us) server_negotiation.cc:407] Connection header received
0620 18:34:30.877144 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.877150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.877379 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.877574 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:30.878366 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.880074 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.882512 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.883956 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.884083 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.884765 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.885681 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.885820 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":879,"threads_started":1}
I20260620 18:34:30.888012 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.872952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.97.2:37055)
0620 18:34:30.877112 (+ 4160us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.877120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.877147 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:34:30.877247 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.877256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.877446 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.877864 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:30.878496 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.880062 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.882649 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.884242 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.887234 (+ 2992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.887427 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.887499 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.887698 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4002,"thread_start_us":3886,"threads_started":1}
I20260620 18:34:30.888447 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 3be276aafeb44678a9b206d4184def9b (DEFAULT_TABLE table=test-table [id=a2b47965d2b34943a9801ecd8c51f565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:30.889963 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be276aafeb44678a9b206d4184def9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:30.889775 20100 tablet_service.cc:1467] Processing CreateTablet for tablet 3be276aafeb44678a9b206d4184def9b (DEFAULT_TABLE table=test-table [id=a2b47965d2b34943a9801ecd8c51f565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:30.891196 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be276aafeb44678a9b206d4184def9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:30.891278 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 3be276aafeb44678a9b206d4184def9b (DEFAULT_TABLE table=test-table [id=a2b47965d2b34943a9801ecd8c51f565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:30.892622 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be276aafeb44678a9b206d4184def9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:30.902267 20184 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Bootstrap starting.
I20260620 18:34:30.905258 20184 tablet_bootstrap.cc:654] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:30.906498 20184 log.cc:824] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:30.906549 20186 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Bootstrap starting.
I20260620 18:34:30.908631 20185 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Bootstrap starting.
I20260620 18:34:30.910099 20184 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: No bootstrap required, opened a new log
I20260620 18:34:30.910377 20184 ts_tablet_manager.cc:1397] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:34:30.910634 20186 tablet_bootstrap.cc:654] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:30.911904 20186 log.cc:824] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:30.912117 20185 tablet_bootstrap.cc:654] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:30.913175 20185 log.cc:824] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:30.915329 20186 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: No bootstrap required, opened a new log
I20260620 18:34:30.915683 20186 ts_tablet_manager.cc:1397] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:34:30.918133 20185 tablet_bootstrap.cc:492] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: No bootstrap required, opened a new log
I20260620 18:34:30.918386 20185 ts_tablet_manager.cc:1397] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:34:30.918740 20184 raft_consensus.cc:348] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.919098 20184 raft_consensus.cc:374] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:30.919234 20184 raft_consensus.cc:729] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8b57ba2b754b4abc90b8c56d643208, State: Initialized, Role: FOLLOWER
I20260620 18:34:30.919948 20184 consensus_queue.cc:260] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.921645 20184 ts_tablet_manager.cc:1428] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 18:34:30.926182 20186 raft_consensus.cc:348] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.926641 20186 raft_consensus.cc:374] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:30.926766 20186 raft_consensus.cc:729] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53bd4a483554190a728c64fb4792913, State: Initialized, Role: FOLLOWER
I20260620 18:34:30.927397 20186 consensus_queue.cc:260] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.928555 20031 heartbeater.cc:503] Master 127.0.97.62:42685 was elected leader, sending a full tablet report...
I20260620 18:34:30.928519 20185 raft_consensus.cc:348] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.928910 20185 raft_consensus.cc:374] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:30.929028 20185 raft_consensus.cc:729] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aabed634e2e4752b091a1862ca6ac81, State: Initialized, Role: FOLLOWER
I20260620 18:34:30.929123 20186 ts_tablet_manager.cc:1428] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:34:30.929622 20185 consensus_queue.cc:260] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.930603 20165 heartbeater.cc:503] Master 127.0.97.62:42685 was elected leader, sending a full tablet report...
I20260620 18:34:30.931288 20185 ts_tablet_manager.cc:1428] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:34:30.932547 20192 raft_consensus.cc:482] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:30.932803 20192 raft_consensus.cc:504] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.936452 20192 leader_election.cc:288] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53bd4a483554190a728c64fb4792913 (127.0.97.2:37055), 3b8b57ba2b754b4abc90b8c56d643208 (127.0.97.1:33915)
I20260620 18:34:30.941274 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.934918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37055 (local address 127.0.97.3:60669)
0620 18:34:30.935372 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.935392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.935539 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.936024 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.936028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.936053 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:30.936296 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.936310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.937374 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.937380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.939129 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.939139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.940711 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.940716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.940875 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.940900 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.940979 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.941066 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":1}
I20260620 18:34:30.941401 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.935060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60669 (local address 127.0.97.2:37055)
0620 18:34:30.935776 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.935780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.935795 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:34:30.935847 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.935850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.935900 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.936000 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:30.936450 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.937242 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.939286 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.940565 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.940697 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.940785 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.941025 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.941144 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582}
I20260620 18:34:30.942265 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be276aafeb44678a9b206d4184def9b" candidate_uuid: "9aabed634e2e4752b091a1862ca6ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53bd4a483554190a728c64fb4792913" is_pre_election: true
I20260620 18:34:30.942761 19986 raft_consensus.cc:2454] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aabed634e2e4752b091a1862ca6ac81 in term 0.
I20260620 18:34:30.943480 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.935449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42239 (local address 127.0.97.1:33915)
0620 18:34:30.935672 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.935679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.937672 (+ 1993us) server_negotiation.cc:407] Connection header received
0620 18:34:30.937820 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.937823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.937884 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.937950 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:30.938574 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.939244 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.941546 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.942456 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.942575 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.942673 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.943102 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.943244 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:34:30.943436 20055 leader_election.cc:302] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aabed634e2e4752b091a1862ca6ac81, a53bd4a483554190a728c64fb4792913; no voters:
I20260620 18:34:30.943645 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.934990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33915 (local address 127.0.97.3:42239)
0620 18:34:30.937570 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:30.937588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:30.937710 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:30.938003 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:30.938009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:30.938036 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:30.938419 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.938430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.939349 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.939354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.941356 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:30.941370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.942703 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.942710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:30.942903 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.942955 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:34:30.943051 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:34:30.943202 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2463,"mutex_wait_us":22,"thread_start_us":1256,"threads_started":1}
I20260620 18:34:30.943945 20192 raft_consensus.cc:2781] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:30.944105 20192 raft_consensus.cc:482] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:30.944211 20192 raft_consensus.cc:3037] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:30.944568 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be276aafeb44678a9b206d4184def9b" candidate_uuid: "9aabed634e2e4752b091a1862ca6ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8b57ba2b754b4abc90b8c56d643208" is_pre_election: true
I20260620 18:34:30.945024 19852 raft_consensus.cc:2454] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aabed634e2e4752b091a1862ca6ac81 in term 0.
I20260620 18:34:30.946779 20192 raft_consensus.cc:504] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.947353 20192 leader_election.cc:288] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [CANDIDATE]: Term 1 election: Requested vote from peers a53bd4a483554190a728c64fb4792913 (127.0.97.2:37055), 3b8b57ba2b754b4abc90b8c56d643208 (127.0.97.1:33915)
I20260620 18:34:30.947855 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be276aafeb44678a9b206d4184def9b" candidate_uuid: "9aabed634e2e4752b091a1862ca6ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53bd4a483554190a728c64fb4792913"
I20260620 18:34:30.947901 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be276aafeb44678a9b206d4184def9b" candidate_uuid: "9aabed634e2e4752b091a1862ca6ac81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8b57ba2b754b4abc90b8c56d643208"
I20260620 18:34:30.948051 19986 raft_consensus.cc:3037] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:30.948072 19852 raft_consensus.cc:3037] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:30.951202 19852 raft_consensus.cc:2454] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aabed634e2e4752b091a1862ca6ac81 in term 1.
I20260620 18:34:30.951202 19986 raft_consensus.cc:2454] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aabed634e2e4752b091a1862ca6ac81 in term 1.
I20260620 18:34:30.951599 20056 leader_election.cc:302] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8b57ba2b754b4abc90b8c56d643208, 9aabed634e2e4752b091a1862ca6ac81; no voters:
I20260620 18:34:30.951967 20192 raft_consensus.cc:2781] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:30.952821 20192 raft_consensus.cc:686] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 1 LEADER]: Becoming Leader. State: Replica: 9aabed634e2e4752b091a1862ca6ac81, State: Running, Role: LEADER
I20260620 18:34:30.953311 20192 consensus_queue.cc:237] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } }
I20260620 18:34:30.957846 19708 catalog_manager.cc:5495] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aabed634e2e4752b091a1862ca6ac81 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9aabed634e2e4752b091a1862ca6ac81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aabed634e2e4752b091a1862ca6ac81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:30.983143 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:30.977950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.0.97.3:41431)
0620 18:34:30.978119 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:30.978124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:30.978177 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:34:30.978358 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:30.978362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:30.978472 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:30.978596 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:30.978913 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.979713 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.981142 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:30.981530 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:30.981623 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:30.982077 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:30.982592 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:30.982702 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:30.982876 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:34:30.982940 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 18:34:30.987573 19898 tablet.cc:2369] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:30.991369 19852 raft_consensus.cc:1261] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 1 FOLLOWER]: Refusing update from remote peer 9aabed634e2e4752b091a1862ca6ac81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:30.991369 19986 raft_consensus.cc:1261] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 1 FOLLOWER]: Refusing update from remote peer 9aabed634e2e4752b091a1862ca6ac81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:30.992532 20194 consensus_queue.cc:1035] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8b57ba2b754b4abc90b8c56d643208" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:30.992923 20192 consensus_queue.cc:1035] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53bd4a483554190a728c64fb4792913" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:31.004240 20197 mvcc.cc:205] Tried to move back new op lower bound from 7298992009169932288 to 7298992009028104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298992009169932288}]
I20260620 18:34:31.004853 20199 mvcc.cc:205] Tried to move back new op lower bound from 7298992009169932288 to 7298992009028104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298992009169932288}]
I20260620 18:34:31.005578 20198 mvcc.cc:205] Tried to move back new op lower bound from 7298992009169932288 to 7298992009028104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298992009169932288}]
I20260620 18:34:31.019205 19708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:31.019541 19708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:31.023231 19708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f7d541a03d74d13aea1cce64edc16ab: Sending DeleteTablet for 3 replicas of tablet 3be276aafeb44678a9b206d4184def9b
I20260620 18:34:31.024487 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be276aafeb44678a9b206d4184def9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52892
I20260620 18:34:31.024672 20100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be276aafeb44678a9b206d4184def9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47382
I20260620 18:34:31.025346 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be276aafeb44678a9b206d4184def9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41106
I20260620 18:34:31.026365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19769
I20260620 18:34:31.034610 20208 tablet_replica.cc:330] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: stopping tablet replica
I20260620 18:34:31.035188 20208 raft_consensus.cc:2229] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:31.036043 20208 raft_consensus.cc:2258] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:31.039309 20207 tablet_replica.cc:330] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: stopping tablet replica
I20260620 18:34:31.039793 20207 raft_consensus.cc:2229] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:31.040019 20209 tablet_replica.cc:330] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: stopping tablet replica
I20260620 18:34:31.040340 20207 raft_consensus.cc:2258] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:31.040417 20209 raft_consensus.cc:2229] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:31.040935 20209 raft_consensus.cc:2258] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:31.042773 20208 ts_tablet_manager.cc:1905] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:31.044684 20209 ts_tablet_manager.cc:1905] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:31.050448 20207 ts_tablet_manager.cc:1905] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:31.051846 20209 ts_tablet_manager.cc:1918] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:31.052063 20209 log.cc:1196] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-0/wal/wals/3be276aafeb44678a9b206d4184def9b
I20260620 18:34:31.052712 20209 ts_tablet_manager.cc:1939] T 3be276aafeb44678a9b206d4184def9b P 3b8b57ba2b754b4abc90b8c56d643208: Deleting consensus metadata
Jun 20 18:34:31 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 18:34:31.054734 19695 catalog_manager.cc:4841] TS 3b8b57ba2b754b4abc90b8c56d643208 (127.0.97.1:33915): tablet 3be276aafeb44678a9b206d4184def9b (table test-table [id=a2b47965d2b34943a9801ecd8c51f565]) successfully deleted
I20260620 18:34:31.058315 20207 ts_tablet_manager.cc:1918] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:31.058493 20207 log.cc:1196] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/3be276aafeb44678a9b206d4184def9b
I20260620 18:34:31.059017 20207 ts_tablet_manager.cc:1939] T 3be276aafeb44678a9b206d4184def9b P a53bd4a483554190a728c64fb4792913: Deleting consensus metadata
I20260620 18:34:31.060953 19694 catalog_manager.cc:4841] TS a53bd4a483554190a728c64fb4792913 (127.0.97.2:37055): tablet 3be276aafeb44678a9b206d4184def9b (table test-table [id=a2b47965d2b34943a9801ecd8c51f565]) successfully deleted
I20260620 18:34:31.061808 20208 ts_tablet_manager.cc:1918] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:31.061991 20208 log.cc:1196] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/3be276aafeb44678a9b206d4184def9b
I20260620 18:34:31.062434 20208 ts_tablet_manager.cc:1939] T 3be276aafeb44678a9b206d4184def9b P 9aabed634e2e4752b091a1862ca6ac81: Deleting consensus metadata
I20260620 18:34:31.063897 19694 catalog_manager.cc:4841] TS 9aabed634e2e4752b091a1862ca6ac81 (127.0.97.3:41431): tablet 3be276aafeb44678a9b206d4184def9b (table test-table [id=a2b47965d2b34943a9801ecd8c51f565]) successfully deleted
I20260620 18:34:31.066263 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:31.028914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.97.1:33915)
0620 18:34:31.029157 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:31.029167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:31.039037 (+ 9870us) server_negotiation.cc:407] Connection header received
0620 18:34:31.041340 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:31.041346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:31.041457 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:31.041542 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:31.044249 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.045498 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.047744 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.048562 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.048689 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:31.054863 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:31.054877 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:31.054896 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:31.054947 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:31.059002 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.063718 (+ 4716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.063729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.063733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.063899 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.064398 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.064404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.064407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.064864 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:31.065183 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:31.065605 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 18:34:31.065843 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:34:31.066933 19892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:31.181994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19769
I20260620 18:34:31.200129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19903
Jun 20 18:34:31 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 18:34:31.220443 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:31.201041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.97.2:37055)
0620 18:34:31.201512 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:31.201516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:31.201533 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:34:31.201590 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:31.201594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:31.201671 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:31.201808 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:31.203797 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.205131 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.207495 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.208321 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.208521 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:31.213742 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:31.213760 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:31.213781 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:31.213825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:31.217557 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.218251 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.218260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.218264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.218408 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.218745 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.218750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.218754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.219234 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:31.219555 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:31.219844 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:34:31.220072 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260620 18:34:31.221134 20026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:31.312397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19903
I20260620 18:34:31.330502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20037
Jun 20 18:34:31 dist-test-slave-mcnm krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 18:34:31.346917 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:31.331457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.97.3:41431)
0620 18:34:31.331692 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:31.331704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:31.331726 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:34:31.331798 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:31.331803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:31.331862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:31.331977 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:31.333253 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.334121 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.335405 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:31.335977 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:31.336130 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:31.340625 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:31.340637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:31.340652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:31.340701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:31.343978 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.344714 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.344727 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.344730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.344917 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:31.345338 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:31.345343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:31.345346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:31.345690 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:31.346023 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:31.346316 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:34:31.346552 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 18:34:31.347726 20160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:31.471482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20037
I20260620 18:34:31.491976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 19676
I20260620 18:34:31.493076 19738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:31.603945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 19676
2026-06-20T18:34:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4331 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:31Z Disabled control of system clock
I20260620 18:34:31.680748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35791
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:32.014575 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:32.014882 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:32.015174 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:32.022985 20218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:32.023113 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:32.023186 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:32.023283 20218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:32.023379 20218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:32.033082 20218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:32.034420 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:32.036382 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:32.048514 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:32.048534 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:32.048995 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:32.049497 20218 server_base.cc:1034] running on GCE node
I20260620 18:34:32.050293 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:32.051751 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:32.052989 20218 hybrid_clock.cc:648] HybridClock initialized: now 1781980472052951 us; error 60 us; skew 500 ppm
I20260620 18:34:32.053339 20218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:32.055706 20218 webserver.cc:460] Webserver started at http://127.0.97.62:34267/ using document root <none> and password file <none>
I20260620 18:34:32.056376 20218 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:32.056509 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:32.056823 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:32.059736 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "515c22bc2dab46e580d5dfb1fb53b60b"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6fecceceb095a8bdbf12b739acc3ac72"
server_key_iv: "67059eb18ccb48748dc06c94eeb42043"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.060539 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "515c22bc2dab46e580d5dfb1fb53b60b"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6fecceceb095a8bdbf12b739acc3ac72"
server_key_iv: "67059eb18ccb48748dc06c94eeb42043"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.065519 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:34:32.068907 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:32.070284 20218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:32.070470 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "515c22bc2dab46e580d5dfb1fb53b60b"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6fecceceb095a8bdbf12b739acc3ac72"
server_key_iv: "67059eb18ccb48748dc06c94eeb42043"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.070677 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:32.093420 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:32.094259 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:32.094554 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:32.113112 20218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35791
I20260620 18:34:32.113121 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35791 every 8 connection(s)
I20260620 18:34:32.114426 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:32.118219 20284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:32.124114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20218
I20260620 18:34:32.124305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:32.124732 388 external_mini_cluster.cc:1428] Setting key 45c6e4e49abf829795389d1386e98658
I20260620 18:34:32.129302 20284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: Bootstrap starting.
I20260620 18:34:32.132463 20284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:32.133430 20284 log.cc:824] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:32.136322 20284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: No bootstrap required, opened a new log
I20260620 18:34:32.138092 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:32.126713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.97.62:35791)
0620 18:34:32.127792 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:32.127803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:32.127845 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:34:32.129546 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:32.129571 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:32.130005 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:32.130383 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:32.130939 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.132265 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.133736 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.134440 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.134636 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:32.135909 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:32.135917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:32.135929 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:32.136460 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:32.136553 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:32.136574 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:32.136716 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:32.136913 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:34:32.137206 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":348,"threads_started":1}
I20260620 18:34:32.144682 20284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } }
I20260620 18:34:32.145083 20284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:32.145195 20284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515c22bc2dab46e580d5dfb1fb53b60b, State: Initialized, Role: FOLLOWER
I20260620 18:34:32.145792 20284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } }
I20260620 18:34:32.146031 20284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:32.146201 20284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:32.146384 20284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:32.148909 20284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } }
I20260620 18:34:32.149396 20284 leader_election.cc:302] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515c22bc2dab46e580d5dfb1fb53b60b; no voters:
I20260620 18:34:32.150068 20284 leader_election.cc:288] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:32.150256 20289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:32.151609 20289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [term 1 LEADER]: Becoming Leader. State: Replica: 515c22bc2dab46e580d5dfb1fb53b60b, State: Running, Role: LEADER
I20260620 18:34:32.152112 20289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } }
I20260620 18:34:32.152411 20284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:32.155112 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515c22bc2dab46e580d5dfb1fb53b60b. Latest consensus state: current_term: 1 leader_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } } }
I20260620 18:34:32.155234 20290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515c22bc2dab46e580d5dfb1fb53b60b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35791 } } }
I20260620 18:34:32.156031 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:32.156126 20290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:32.160597 20298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:32.165519 20298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:32.174163 20298 catalog_manager.cc:1347] Generated new cluster ID: d752b3bdd4964307b30b35b91a643861
I20260620 18:34:32.174281 20298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:32.183279 20298 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:32.184571 20298 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:32.193166 20298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: Generated new TSK 0
I20260620 18:34:32.193979 20298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:32.208022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--builtin_ntp_servers=127.0.97.20:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:34:32.539984 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:32.540262 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:32.540351 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:32.547605 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:32.547706 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:32.547811 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 18:34:32.557083 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:32.558502 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:32.560305 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:32.573429 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:32.573421 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:32.573475 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:32.574195 20308 server_base.cc:1034] running on GCE node
I20260620 18:34:32.574810 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:32.575800 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:32.576998 20308 hybrid_clock.cc:648] HybridClock initialized: now 1781980472576957 us; error 35 us; skew 500 ppm
I20260620 18:34:32.577346 20308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:32.579658 20308 webserver.cc:460] Webserver started at http://127.0.97.1:33973/ using document root <none> and password file <none>
I20260620 18:34:32.580348 20308 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:32.580459 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:32.580794 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:32.583621 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data/instance:
uuid: "6479190ddde94413a4c7da4fd8e2e035"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6f3c1362c321678921bf2f60883387ba"
server_key_iv: "924bd670ca3832e4e57990a08b5a56df"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.584399 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6479190ddde94413a4c7da4fd8e2e035"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6f3c1362c321678921bf2f60883387ba"
server_key_iv: "924bd670ca3832e4e57990a08b5a56df"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.589206 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:34:32.592566 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:32.593746 20308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:32.593909 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal
uuid: "6479190ddde94413a4c7da4fd8e2e035"
format_stamp: "Formatted at 2026-06-20 18:34:32 on dist-test-slave-mcnm"
server_key: "6f3c1362c321678921bf2f60883387ba"
server_key_iv: "924bd670ca3832e4e57990a08b5a56df"
server_key_version: "encryptionkey@0"
I20260620 18:34:32.594122 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:32.607041 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:32.607818 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:32.608088 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:32.609005 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:32.610414 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:32.610527 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:32.610664 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:32.610743 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:32.645067 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32961
I20260620 18:34:32.645115 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32961 every 8 connection(s)
I20260620 18:34:32.646453 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:32.652176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20308
I20260620 18:34:32.652359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:32.652848 388 external_mini_cluster.cc:1428] Setting key 45163948e90b4da30b95054aa219ad90
I20260620 18:34:32.655437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--builtin_ntp_servers=127.0.97.20:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:32.659296 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:32.648936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35383 (local address 127.0.97.62:35791)
0620 18:34:32.649380 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:32.649386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:32.650593 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 18:34:32.652634 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:32.652640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:32.652772 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:32.652898 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:32.653968 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.654705 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.656471 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.656909 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.657008 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:32.658554 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:32.658559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:32.658562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:32.658580 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:32.658622 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:32.658635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:32.658711 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:32.658954 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:34:32.659032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":315,"thread_start_us":204,"threads_started":1}
I20260620 18:34:32.660691 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:32.649550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35791 (local address 127.0.97.1:35383)
0620 18:34:32.650428 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:32.650469 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:32.652310 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:32.653137 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:32.653145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:32.653238 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:32.653780 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:32.653796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.654860 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.654866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:32.656250 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:32.656262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:32.657127 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:32.657137 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:32.657494 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:32.658235 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:32.658241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:32.658289 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:32.658383 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:32.658733 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:32.658746 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:32.658756 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:34:32.658926 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:34:32.659316 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":275,"threads_started":1}
I20260620 18:34:32.662178 20435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35791
I20260620 18:34:32.662480 20435 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:32.663197 20435 heartbeater.cc:511] Master 127.0.97.62:35791 requested a full tablet report, sending...
I20260620 18:34:32.665215 20249 ts_manager.cc:194] Registered new tserver with Master: 6479190ddde94413a4c7da4fd8e2e035 (127.0.97.1:32961)
I20260620 18:34:32.666632 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35383
W20260620 18:34:32.983063 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:32.983335 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:32.983484 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:32.990799 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:32.990890 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:32.991055 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 18:34:33.000218 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:33.001490 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:33.003293 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:33.016244 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.016286 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.016386 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:33.017324 20439 server_base.cc:1034] running on GCE node
I20260620 18:34:33.017949 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:33.018919 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:33.020193 20439 hybrid_clock.cc:648] HybridClock initialized: now 1781980473020137 us; error 61 us; skew 500 ppm
I20260620 18:34:33.020541 20439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:33.022941 20439 webserver.cc:460] Webserver started at http://127.0.97.2:45427/ using document root <none> and password file <none>
I20260620 18:34:33.023700 20439 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:33.023829 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:33.024142 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:33.027006 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data/instance:
uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "6d654df691fd588fb019efb8f6941a09"
server_key_iv: "45a7721593815b1180ba4bdedf14b36f"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.027864 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "6d654df691fd588fb019efb8f6941a09"
server_key_iv: "45a7721593815b1180ba4bdedf14b36f"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.032886 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:33.036119 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.037309 20439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 18:34:33.037475 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal
uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "6d654df691fd588fb019efb8f6941a09"
server_key_iv: "45a7721593815b1180ba4bdedf14b36f"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.037679 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:33.060804 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:33.061575 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:33.061851 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:33.062790 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:33.064301 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:33.064425 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.064558 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:33.064644 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.097765 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36711
I20260620 18:34:33.097851 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36711 every 8 connection(s)
I20260620 18:34:33.099064 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:33.101460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20439
I20260620 18:34:33.101630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:33.102183 388 external_mini_cluster.cc:1428] Setting key 474f67dcbbd772a59a33c592dcbe3023
I20260620 18:34:33.104908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--builtin_ntp_servers=127.0.97.20:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:34:33.112553 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.101757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52999 (local address 127.0.97.62:35791)
0620 18:34:33.101974 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.101982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.103472 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 18:34:33.105319 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.105328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.105417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.105525 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:33.106538 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.107679 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.109318 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.110059 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.110208 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.111509 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:33.111547 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:33.111555 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:33.111581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:33.111622 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:33.111640 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:33.111734 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.111998 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.112111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:34:33.113636 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.102223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35791 (local address 127.0.97.2:52999)
0620 18:34:33.103306 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.103351 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.105011 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.105742 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.105750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.105841 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:33.106371 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.106386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.107856 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.107860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.109168 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.109182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.110277 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.110282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.110502 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.111209 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:33.111212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:33.111249 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:33.111328 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:33.111771 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:33.111792 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:33.111799 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.111959 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.112309 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":347,"threads_started":1}
I20260620 18:34:33.115177 20566 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35791
I20260620 18:34:33.115494 20566 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:33.116415 20566 heartbeater.cc:511] Master 127.0.97.62:35791 requested a full tablet report, sending...
I20260620 18:34:33.118034 20249 ts_manager.cc:194] Registered new tserver with Master: 5b67e4df8c7f4056a3d3ac888d96a1ff (127.0.97.2:36711)
I20260620 18:34:33.119151 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52999
W20260620 18:34:33.451208 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:33.451532 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:33.451709 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:33.459156 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:33.459286 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:33.459468 20569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 18:34:33.468488 20569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:33.469771 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:33.471653 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:33.484485 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.484625 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.484997 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:33.485567 20569 server_base.cc:1034] running on GCE node
I20260620 18:34:33.486156 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:33.487077 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:33.488307 20569 hybrid_clock.cc:648] HybridClock initialized: now 1781980473488258 us; error 73 us; skew 500 ppm
I20260620 18:34:33.488634 20569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:34:33.490906 20569 webserver.cc:460] Webserver started at http://127.0.97.3:43873/ using document root <none> and password file <none>
I20260620 18:34:33.491655 20569 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:33.491786 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:33.492100 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:33.494866 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data/instance:
uuid: "5db68fca3aaf4293bdd886562f1a2488"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "3fdb667ca8467e70f3b5cd93d9be59ce"
server_key_iv: "cbcc199da56c7fd341f2c69e4da86e94"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.495725 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5db68fca3aaf4293bdd886562f1a2488"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "3fdb667ca8467e70f3b5cd93d9be59ce"
server_key_iv: "cbcc199da56c7fd341f2c69e4da86e94"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.500777 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:34:33.504115 20583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.505409 20569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:33.505595 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal
uuid: "5db68fca3aaf4293bdd886562f1a2488"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-mcnm"
server_key: "3fdb667ca8467e70f3b5cd93d9be59ce"
server_key_iv: "cbcc199da56c7fd341f2c69e4da86e94"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.505811 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:33.550505 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:33.551369 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:33.551707 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:33.552680 20569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:33.554319 20569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:33.554447 20569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.554585 20569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:33.554673 20569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.588475 20569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35703
I20260620 18:34:33.588572 20695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35703 every 8 connection(s)
I20260620 18:34:33.590092 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:33.594163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20569
I20260620 18:34:33.594334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:33.594892 388 external_mini_cluster.cc:1428] Setting key 15f14c56826c545ad99fe7b9f39473e4
I20260620 18:34:33.603248 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.592700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59709 (local address 127.0.97.62:35791)
0620 18:34:33.592933 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.592940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.594592 (+ 1652us) server_negotiation.cc:407] Connection header received
0620 18:34:33.597030 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.597036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.597114 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.597222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:33.598439 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.599175 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.600857 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.601273 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.601368 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.602626 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:33.602630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:33.602633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:33.602649 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:33.602676 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:33.602687 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:33.602752 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.602946 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.603020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:34:33.604254 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.593323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35791 (local address 127.0.97.3:59709)
0620 18:34:33.594420 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.594470 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.596734 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.597448 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.597459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.597598 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:33.598283 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.598300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.599317 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.599323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.600686 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.600698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.601451 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.601456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.601645 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.602366 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:33.602369 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:33.602406 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:33.602478 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:33.602769 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:33.602776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:33.602781 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.602944 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.603212 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":277,"threads_started":1}
I20260620 18:34:33.605549 20696 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35791
I20260620 18:34:33.605870 20696 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:33.606602 20696 heartbeater.cc:511] Master 127.0.97.62:35791 requested a full tablet report, sending...
I20260620 18:34:33.608014 20249 ts_manager.cc:194] Registered new tserver with Master: 5db68fca3aaf4293bdd886562f1a2488 (127.0.97.3:35703)
I20260620 18:34:33.609211 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59709
I20260620 18:34:33.610260 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:33.620796 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.614652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.97.62:35791)
0620 18:34:33.614821 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.614826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.615250 (+ 424us) server_negotiation.cc:407] Connection header received
0620 18:34:33.615373 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.615377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.615450 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.615601 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:33.616284 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.617024 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.618585 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.618999 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.619107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.620196 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:33.620200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:33.620204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:33.620220 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:33.620251 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:33.620263 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:33.620342 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.620505 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.620579 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:34:33.625101 20248 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:33.631932 20248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:33.669453 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.643714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60832)
0620 18:34:33.655700 (+ 11986us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.655762 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.656008 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.660435 (+ 4427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.660445 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.660484 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:33.660947 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.660979 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.663559 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.663568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.665981 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.665992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.668270 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.668276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.668512 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.668696 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.668825 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.669037 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11625,"mutex_wait_us":111,"thread_start_us":11459,"threads_started":1}
I20260620 18:34:33.669404 20435 heartbeater.cc:503] Master 127.0.97.62:35791 was elected leader, sending a full tablet report...
I20260620 18:34:33.669477 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.643976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.97.1:32961)
0620 18:34:33.659544 (+ 15568us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.659556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.659592 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:34:33.659712 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.659722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.660011 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.661204 (+ 1193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:33.661451 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.663363 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.666251 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.667723 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.667857 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.668139 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.668902 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.669083 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15347,"thread_start_us":15225,"threads_started":1}
I20260620 18:34:33.673023 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.656430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35703 (local address 127.0.0.1:51872)
0620 18:34:33.665129 (+ 8699us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.665146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.665329 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.665955 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.665960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.665984 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:33.666387 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.666398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.668064 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.668069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.670243 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.670253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.672261 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.672267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.672431 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.672485 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.672583 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.672701 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8541,"thread_start_us":3740,"threads_started":1}
I20260620 18:34:33.673141 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.656585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.0.97.3:35703)
0620 18:34:33.660673 (+ 4088us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.660679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.665234 (+ 4555us) server_negotiation.cc:407] Connection header received
0620 18:34:33.665538 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.665546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.665757 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.665954 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:33.666569 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.667902 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.670419 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.672073 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.672233 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.672503 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.672622 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.672768 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3933,"thread_start_us":3782,"threads_started":1}
I20260620 18:34:33.673730 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.644385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36711 (local address 127.0.0.1:35422)
0620 18:34:33.659947 (+ 15562us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.659974 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.660120 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.664532 (+ 4412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.664540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.664576 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:33.665221 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.665237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.666970 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.666975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.669053 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.669067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.672747 (+ 3680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.672756 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.672908 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.672944 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.673013 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.673448 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15378,"thread_start_us":20430,"threads_started":1}
I20260620 18:34:33.673828 20370 tablet_service.cc:1467] Processing CreateTablet for tablet 9953d87191c54dbdadaf2820d267b09a (DEFAULT_TABLE table=test-table [id=31f30bf4ee644e63bbce745caf0eb996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:33.674827 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.645086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.2:36711)
0620 18:34:33.663951 (+ 18865us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.663967 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.663996 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:34:33.664091 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.664100 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.664323 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.664522 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:33.665414 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.666830 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.671145 (+ 4315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.672529 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.674061 (+ 1532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.674288 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.674353 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.674514 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18703,"thread_start_us":18598,"threads_started":1}
I20260620 18:34:33.675400 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9953d87191c54dbdadaf2820d267b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:33.676565 20631 tablet_service.cc:1467] Processing CreateTablet for tablet 9953d87191c54dbdadaf2820d267b09a (DEFAULT_TABLE table=test-table [id=31f30bf4ee644e63bbce745caf0eb996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:33.677947 20631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9953d87191c54dbdadaf2820d267b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:33.678141 20501 tablet_service.cc:1467] Processing CreateTablet for tablet 9953d87191c54dbdadaf2820d267b09a (DEFAULT_TABLE table=test-table [id=31f30bf4ee644e63bbce745caf0eb996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:33.679399 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9953d87191c54dbdadaf2820d267b09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:33.688047 20714 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Bootstrap starting.
I20260620 18:34:33.690747 20714 tablet_bootstrap.cc:654] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:33.692054 20714 log.cc:824] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:33.692148 20715 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Bootstrap starting.
I20260620 18:34:33.693642 20716 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Bootstrap starting.
I20260620 18:34:33.696471 20715 tablet_bootstrap.cc:654] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:33.696950 20716 tablet_bootstrap.cc:654] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:33.697710 20715 log.cc:824] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:33.697994 20716 log.cc:824] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:33.698376 20714 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: No bootstrap required, opened a new log
I20260620 18:34:33.698689 20714 ts_tablet_manager.cc:1397] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 18:34:33.700925 20715 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: No bootstrap required, opened a new log
I20260620 18:34:33.701192 20715 ts_tablet_manager.cc:1397] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 18:34:33.704875 20716 tablet_bootstrap.cc:492] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: No bootstrap required, opened a new log
I20260620 18:34:33.705155 20716 ts_tablet_manager.cc:1397] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260620 18:34:33.709429 20714 raft_consensus.cc:348] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.709939 20714 raft_consensus.cc:374] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:33.710037 20714 raft_consensus.cc:729] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6479190ddde94413a4c7da4fd8e2e035, State: Initialized, Role: FOLLOWER
I20260620 18:34:33.710702 20714 consensus_queue.cc:260] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.711477 20715 raft_consensus.cc:348] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.711912 20715 raft_consensus.cc:374] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:33.712026 20715 raft_consensus.cc:729] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db68fca3aaf4293bdd886562f1a2488, State: Initialized, Role: FOLLOWER
I20260620 18:34:33.712616 20714 ts_tablet_manager.cc:1428] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260620 18:34:33.712709 20715 consensus_queue.cc:260] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.713093 20716 raft_consensus.cc:348] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.713496 20716 raft_consensus.cc:374] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:33.713601 20716 raft_consensus.cc:729] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b67e4df8c7f4056a3d3ac888d96a1ff, State: Initialized, Role: FOLLOWER
I20260620 18:34:33.714395 20715 ts_tablet_manager.cc:1428] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:34:33.714260 20716 consensus_queue.cc:260] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.714954 20696 heartbeater.cc:503] Master 127.0.97.62:35791 was elected leader, sending a full tablet report...
I20260620 18:34:33.715703 20566 heartbeater.cc:503] Master 127.0.97.62:35791 was elected leader, sending a full tablet report...
I20260620 18:34:33.716343 20716 ts_tablet_manager.cc:1428] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260620 18:34:33.813719 20722 raft_consensus.cc:482] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:33.814067 20722 raft_consensus.cc:504] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.815814 20722 leader_election.cc:288] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6479190ddde94413a4c7da4fd8e2e035 (127.0.97.1:32961), 5db68fca3aaf4293bdd886562f1a2488 (127.0.97.3:35703)
I20260620 18:34:33.822571 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.816354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42079 (local address 127.0.97.1:32961)
0620 18:34:33.816628 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.816635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.816952 (+ 317us) server_negotiation.cc:407] Connection header received
0620 18:34:33.817184 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.817212 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.817321 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.817425 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:33.818221 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.819059 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.820876 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.821659 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.821808 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.821918 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.822204 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.822333 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:34:33.822717 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.816194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.97.2:42079)
0620 18:34:33.816833 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.816851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.817032 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.817501 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.817507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.817537 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:33.818032 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.818045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.819190 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.819198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.820673 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.820685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.821868 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.821875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.822046 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.822070 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.822175 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.822302 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1,"spinlock_wait_cycles":22272,"thread_start_us":338,"threads_started":1}
I20260620 18:34:33.823434 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.817117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46397 (local address 127.0.97.3:35703)
0620 18:34:33.817343 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.817348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.817726 (+ 378us) server_negotiation.cc:407] Connection header received
0620 18:34:33.817974 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.817984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.818046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.818145 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:33.819143 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.819926 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.821609 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.822365 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.822498 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.822604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.823082 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.823209 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:34:33.823467 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.816983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35703 (local address 127.0.97.2:46397)
0620 18:34:33.817568 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:33.817621 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:33.817763 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:33.818574 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:33.818578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:33.818601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:33.818986 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.818995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.820059 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.820065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.821433 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:33.821445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.822569 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.822575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:33.822926 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.822950 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:34:33.823055 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:33.823184 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":291,"threads_started":1}
I20260620 18:34:33.823799 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9953d87191c54dbdadaf2820d267b09a" candidate_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6479190ddde94413a4c7da4fd8e2e035" is_pre_election: true
I20260620 18:34:33.824455 20390 raft_consensus.cc:2454] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b67e4df8c7f4056a3d3ac888d96a1ff in term 0.
I20260620 18:34:33.824492 20651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9953d87191c54dbdadaf2820d267b09a" candidate_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db68fca3aaf4293bdd886562f1a2488" is_pre_election: true
I20260620 18:34:33.825045 20651 raft_consensus.cc:2454] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b67e4df8c7f4056a3d3ac888d96a1ff in term 0.
I20260620 18:34:33.825122 20456 leader_election.cc:302] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b67e4df8c7f4056a3d3ac888d96a1ff, 6479190ddde94413a4c7da4fd8e2e035; no voters:
I20260620 18:34:33.825557 20722 raft_consensus.cc:2781] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:33.825704 20722 raft_consensus.cc:482] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:33.825829 20722 raft_consensus.cc:3037] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:33.828146 20722 raft_consensus.cc:504] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.828765 20722 leader_election.cc:288] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [CANDIDATE]: Term 1 election: Requested vote from peers 6479190ddde94413a4c7da4fd8e2e035 (127.0.97.1:32961), 5db68fca3aaf4293bdd886562f1a2488 (127.0.97.3:35703)
I20260620 18:34:33.829187 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9953d87191c54dbdadaf2820d267b09a" candidate_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6479190ddde94413a4c7da4fd8e2e035"
I20260620 18:34:33.829305 20651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9953d87191c54dbdadaf2820d267b09a" candidate_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db68fca3aaf4293bdd886562f1a2488"
I20260620 18:34:33.829401 20390 raft_consensus.cc:3037] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:33.829528 20651 raft_consensus.cc:3037] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:33.831712 20390 raft_consensus.cc:2454] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b67e4df8c7f4056a3d3ac888d96a1ff in term 1.
I20260620 18:34:33.831776 20651 raft_consensus.cc:2454] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b67e4df8c7f4056a3d3ac888d96a1ff in term 1.
I20260620 18:34:33.832115 20456 leader_election.cc:302] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b67e4df8c7f4056a3d3ac888d96a1ff, 6479190ddde94413a4c7da4fd8e2e035; no voters:
I20260620 18:34:33.832434 20722 raft_consensus.cc:2781] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:33.832955 20722 raft_consensus.cc:686] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 1 LEADER]: Becoming Leader. State: Replica: 5b67e4df8c7f4056a3d3ac888d96a1ff, State: Running, Role: LEADER
I20260620 18:34:33.833426 20722 consensus_queue.cc:237] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } }
I20260620 18:34:33.837885 20247 catalog_manager.cc:5495] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b67e4df8c7f4056a3d3ac888d96a1ff (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b67e4df8c7f4056a3d3ac888d96a1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36711 } health_report { overall_health: HEALTHY } } }
W20260620 18:34:33.841796 20697 tablet.cc:2369] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:33.851306 20567 tablet.cc:2369] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:33.868661 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.863146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.97.2:36711)
0620 18:34:33.863379 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.863385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.863406 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:34:33.863603 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.863607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.863697 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.863852 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:33.864258 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.864992 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.866415 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.866851 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.866990 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.867501 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:33.868080 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:33.868244 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.868298 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.868376 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:34:33.877161 20390 raft_consensus.cc:1261] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 1 FOLLOWER]: Refusing update from remote peer 5b67e4df8c7f4056a3d3ac888d96a1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:33.877159 20651 raft_consensus.cc:1261] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 1 FOLLOWER]: Refusing update from remote peer 5b67e4df8c7f4056a3d3ac888d96a1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:33.878203 20725 consensus_queue.cc:1035] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6479190ddde94413a4c7da4fd8e2e035" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:33.878561 20722 consensus_queue.cc:1035] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db68fca3aaf4293bdd886562f1a2488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:33.891352 20729 mvcc.cc:205] Tried to move back new op lower bound from 7298992020989403136 to 7298992020825063424. Current Snapshot: MvccSnapshot[applied={T|T < 7298992020989403136}]
I20260620 18:34:33.905206 20249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:33.905490 20249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:33.909345 20249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515c22bc2dab46e580d5dfb1fb53b60b: Sending DeleteTablet for 3 replicas of tablet 9953d87191c54dbdadaf2820d267b09a
I20260620 18:34:33.910760 20631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9953d87191c54dbdadaf2820d267b09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:33 UTC) from {username='slave'} at 127.0.0.1:51872
I20260620 18:34:33.910760 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet 9953d87191c54dbdadaf2820d267b09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:33 UTC) from {username='slave'} at 127.0.0.1:60832
I20260620 18:34:33.910997 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet 9953d87191c54dbdadaf2820d267b09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:33 UTC) from {username='slave'} at 127.0.0.1:35422
I20260620 18:34:33.913036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20308
I20260620 18:34:33.923724 20739 tablet_replica.cc:330] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: stopping tablet replica
I20260620 18:34:33.924328 20739 raft_consensus.cc:2229] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:33.925140 20739 raft_consensus.cc:2258] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:33.927965 20739 ts_tablet_manager.cc:1905] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:33.937443 20739 ts_tablet_manager.cc:1918] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:33.937660 20739 log.cc:1196] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-1/wal/wals/9953d87191c54dbdadaf2820d267b09a
I20260620 18:34:33.938313 20739 ts_tablet_manager.cc:1939] T 9953d87191c54dbdadaf2820d267b09a P 5b67e4df8c7f4056a3d3ac888d96a1ff: Deleting consensus metadata
I20260620 18:34:33.940210 20233 catalog_manager.cc:4841] TS 5b67e4df8c7f4056a3d3ac888d96a1ff (127.0.97.2:36711): tablet 9953d87191c54dbdadaf2820d267b09a (table test-table [id=31f30bf4ee644e63bbce745caf0eb996]) successfully deleted
I20260620 18:34:33.943238 20737 tablet_replica.cc:330] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: stopping tablet replica
I20260620 18:34:33.943727 20737 raft_consensus.cc:2229] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:33.944208 20737 raft_consensus.cc:2258] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:33.945636 20738 tablet_replica.cc:330] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: stopping tablet replica
I20260620 18:34:33.945998 20738 raft_consensus.cc:2229] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:33.946420 20738 raft_consensus.cc:2258] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:33.947321 20737 ts_tablet_manager.cc:1905] T 9953d87191c54dbdadaf2820d267b09a P 6479190ddde94413a4c7da4fd8e2e035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:33.948869 20738 ts_tablet_manager.cc:1905] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:33.949484 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.914694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.97.1:32961)
0620 18:34:33.914930 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.914939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.937165 (+ 22226us) server_negotiation.cc:407] Connection header received
0620 18:34:33.937749 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.937756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.937885 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.937973 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:33.938737 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.939886 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.941990 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.942964 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.943123 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.947162 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:33.947174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:33.947191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:33.948124 (+ 933us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:33.948188 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:33.948216 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:33.948335 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.948842 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.949110 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:34:33.950611 20430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:33.958007 20738 ts_tablet_manager.cc:1918] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:33.958225 20738 log.cc:1196] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Deleting WAL directory at /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980152255989-388-0/minicluster-data/ts-2/wal/wals/9953d87191c54dbdadaf2820d267b09a
I20260620 18:34:33.958926 20738 ts_tablet_manager.cc:1939] T 9953d87191c54dbdadaf2820d267b09a P 5db68fca3aaf4293bdd886562f1a2488: Deleting consensus metadata
I20260620 18:34:33.960678 20235 catalog_manager.cc:4841] TS 5db68fca3aaf4293bdd886562f1a2488 (127.0.97.3:35703): tablet 9953d87191c54dbdadaf2820d267b09a (table test-table [id=31f30bf4ee644e63bbce745caf0eb996]) successfully deleted
I20260620 18:34:34.072736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20308
I20260620 18:34:34.090003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20439
W20260620 18:34:34.090000 20235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32961 (error 108)
I20260620 18:34:34.092120 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.091561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60856)
0620 18:34:34.091837 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.091965 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260620 18:34:34.092368 20235 catalog_manager.cc:4584] TS 6479190ddde94413a4c7da4fd8e2e035 (127.0.97.1:32961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9953d87191c54dbdadaf2820d267b09a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
I20260620 18:34:34.101145 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.091029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.97.2:36711)
0620 18:34:34.091206 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:34.091212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:34.091304 (+ 92us) server_negotiation.cc:407] Connection header received
0620 18:34:34.091550 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:34.091559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:34.091678 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:34.091794 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:34.092322 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.093325 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.094588 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.095076 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.095257 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:34.099417 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:34.099435 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:34.099456 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:34.100276 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:34.100340 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:34.100366 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:34.100482 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:34.100657 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:34:34.100771 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:34.101884 20561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:34.146015 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.145459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60864)
0620 18:34:34.145692 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.145805 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:34:34.147164 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.146776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60874)
0620 18:34:34.146944 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.147028 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:34:34.207962 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.207442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60888)
0620 18:34:34.207678 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.207788 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:34:34.209154 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.208677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60890)
0620 18:34:34.208894 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.208991 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 18:34:34.216315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20439
I20260620 18:34:34.235069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20569
I20260620 18:34:34.242446 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.236008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.97.3:35703)
0620 18:34:34.236283 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:34.236291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:34.236312 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:34:34.236375 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:34.236380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:34.236491 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:34.236616 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:34.237132 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.238084 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.239376 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.239931 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.240063 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:34.241111 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:34.241121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:34.241135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:34.241713 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:34.241766 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:34.241786 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:34.241918 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:34.242016 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:34:34.242117 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:34:34.243072 20691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:34.289942 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.289367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60900)
0620 18:34:34.289619 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.289731 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 18:34:34.291299 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.290822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32961 (local address 127.0.0.1:60906)
0620 18:34:34.291056 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.291154 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 18:34:34.331672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20569
I20260620 18:34:34.348457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20218
I20260620 18:34:34.349660 20279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:34.460440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20218
2026-06-20T18:34:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2857 ms)
[----------] 6 tests from EncryptionPolicyTest (21388 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:34 dist-test-slave-mcnm krb5kdc[20748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:34 dist-test-slave-mcnm krb5kdc[20748](info): set up 2 sockets
Jun 20 18:34:34 dist-test-slave-mcnm krb5kdc[20748](info): commencing operation
krb5kdc: starting...
W20260620 18:34:35.595950 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:35 dist-test-slave-mcnm krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:35.855865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:36.192047 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:36.192332 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:36.192430 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:36.199810 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:36.199936 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:36.199985 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:36.200038 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:36.200084 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:36.210049 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:36.211419 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:36.212915 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:36.225360 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.225360 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.225399 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:36.226063 20761 server_base.cc:1034] running on GCE node
I20260620 18:34:36.226787 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:36.228209 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:36.229463 20761 hybrid_clock.cc:648] HybridClock initialized: now 1781980476229427 us; error 48 us; skew 500 ppm
I20260620 18:34:36.229864 20761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:36 dist-test-slave-mcnm krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:36.233004 20761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:36.234391 20761 webserver.cc:460] Webserver started at http://127.0.0.1:44207/ using document root <none> and password file <none>
I20260620 18:34:36.235028 20761 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:36.235128 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:36.235371 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:36.238301 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "a40903b9828b4b2aac83239c25b93ef0"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-mcnm"
server_key: "e621bdbd161e33eb6706e28d9af12f02"
server_key_iv: "a129c85c80bbb6f792a898e80e2f9601"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.239027 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "a40903b9828b4b2aac83239c25b93ef0"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-mcnm"
server_key: "e621bdbd161e33eb6706e28d9af12f02"
server_key_iv: "a129c85c80bbb6f792a898e80e2f9601"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.245112 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 18:34:36.248929 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.250295 20761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:36.250469 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "a40903b9828b4b2aac83239c25b93ef0"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-mcnm"
server_key: "e621bdbd161e33eb6706e28d9af12f02"
server_key_iv: "a129c85c80bbb6f792a898e80e2f9601"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.250667 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:36.267956 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:36.272138 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:36.272421 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:36.290344 20761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43373
I20260620 18:34:36.290376 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43373 every 8 connection(s)
I20260620 18:34:36.291710 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:36.295399 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:36.301082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20761
I20260620 18:34:36.301299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:36.301947 388 external_mini_cluster.cc:1428] Setting key cc0b97973c3419c14d2cc8a7b0db0528
I20260620 18:34:36.306954 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0: Bootstrap starting.
I20260620 18:34:36.310622 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:36.311944 20828 log.cc:824] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:36.315295 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0: No bootstrap required, opened a new log
Jun 20 18:34:36 dist-test-slave-mcnm krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:34:36.323668 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } }
I20260620 18:34:36.324064 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:36.324193 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40903b9828b4b2aac83239c25b93ef0, State: Initialized, Role: FOLLOWER
I20260620 18:34:36.324829 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } }
I20260620 18:34:36.325089 20828 raft_consensus.cc:388] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:36.325244 20828 raft_consensus.cc:482] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:36.325446 20828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:36.327168 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:36.304007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56939 (local address 127.0.0.1:43373)
0620 18:34:36.304903 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:36.304918 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:36.304980 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:34:36.306647 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:36.306667 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:36.307048 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:36.307406 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:36.309302 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.310582 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.312549 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.313568 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.313705 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:36.318555 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:36.318562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:36.318575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:36.318616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:36.322172 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.323080 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.323088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.323094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.323279 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.323878 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.323882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.323885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.324584 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:36.324961 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:36.325430 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 18:34:36.325798 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":429,"threads_started":1}
I20260620 18:34:36.328405 20828 raft_consensus.cc:504] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } }
I20260620 18:34:36.328912 20828 leader_election.cc:302] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a40903b9828b4b2aac83239c25b93ef0; no voters:
I20260620 18:34:36.329497 20828 leader_election.cc:288] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:36.329682 20833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:36.331249 20833 raft_consensus.cc:686] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [term 1 LEADER]: Becoming Leader. State: Replica: a40903b9828b4b2aac83239c25b93ef0, State: Running, Role: LEADER
I20260620 18:34:36.331873 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:36.331974 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:36.331900 20833 consensus_queue.cc:237] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } }
I20260620 18:34:36.341147 20834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a40903b9828b4b2aac83239c25b93ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } } }
I20260620 18:34:36.341944 20834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:36.340924 20835 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a40903b9828b4b2aac83239c25b93ef0. Latest consensus state: current_term: 1 leader_uuid: "a40903b9828b4b2aac83239c25b93ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40903b9828b4b2aac83239c25b93ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43373 } } }
I20260620 18:34:36.343106 20835 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:36.356277 20844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:36.364235 20844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:36.372419 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:36.343730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53131 (local address 127.0.0.1:43373)
0620 18:34:36.343927 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:36.343935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:36.351034 (+ 7099us) server_negotiation.cc:407] Connection header received
0620 18:34:36.351284 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:36.351293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:36.351408 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:36.351561 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:36.354529 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.356206 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.358134 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.360199 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.360342 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:36.362887 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:36.362898 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:36.362906 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:36.362952 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:36.368049 (+ 5097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.369097 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.369102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.369106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.369250 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.369841 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.369847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.369852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.370236 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:36.371650 (+ 1414us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:36.371847 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:34:36.372021 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:34:36.378456 20844 catalog_manager.cc:1347] Generated new cluster ID: 2049ec3a62894fdaa88690cab180b3a5
I20260620 18:34:36.379172 20844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:36.388912 20844 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:36.390836 20844 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:36.411460 20844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a40903b9828b4b2aac83239c25b93ef0: Generated new TSK 0
I20260620 18:34:36.412514 20844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:36.424495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20761
I20260620 18:34:36.425771 20823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:36.517870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20761
W20260620 18:34:36.535027 20746 connection.cc:535] client connection to 127.0.0.1:43373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43373): Connection reset by peer (error 104)
2026-06-20T18:34:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:36 dist-test-slave-mcnm krb5kdc[20860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:36 dist-test-slave-mcnm krb5kdc[20860](info): set up 2 sockets
Jun 20 18:34:36 dist-test-slave-mcnm krb5kdc[20860](info): commencing operation
krb5kdc: starting...
W20260620 18:34:37.627904 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:37 dist-test-slave-mcnm krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:37.882349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:38.218180 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:38.218473 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:38.218647 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:38.225989 20873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:38.226094 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:38.226204 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:38.226293 20873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:38.226372 20873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:38.236276 20873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:38.237562 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:38.238850 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:38.250631 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.250597 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.250682 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.252470 20873 server_base.cc:1034] running on GCE node
I20260620 18:34:38.253288 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:38.254803 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:38.256053 20873 hybrid_clock.cc:648] HybridClock initialized: now 1781980478255998 us; error 42 us; skew 500 ppm
I20260620 18:34:38.256512 20873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:38 dist-test-slave-mcnm krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:38.259745 20873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:38.261210 20873 webserver.cc:460] Webserver started at http://127.0.0.1:35423/ using document root <none> and password file <none>
I20260620 18:34:38.261919 20873 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:38.262071 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:38.262382 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:38.265448 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "7f530dc6c0854395aaf9d73a5a09a9f8"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-mcnm"
server_key: "ec21a0a989030622bf0236f0fc6ed593"
server_key_iv: "9f346033536fcdcee884a25bac6cb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.266232 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f530dc6c0854395aaf9d73a5a09a9f8"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-mcnm"
server_key: "ec21a0a989030622bf0236f0fc6ed593"
server_key_iv: "9f346033536fcdcee884a25bac6cb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.271445 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:34:38.275090 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.276595 20873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:34:38.276785 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "7f530dc6c0854395aaf9d73a5a09a9f8"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-mcnm"
server_key: "ec21a0a989030622bf0236f0fc6ed593"
server_key_iv: "9f346033536fcdcee884a25bac6cb5e5"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.277005 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:38.290138 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:38.294188 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:38.294476 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:38.312225 20873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36691
I20260620 18:34:38.312278 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36691 every 8 connection(s)
I20260620 18:34:38.313627 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:38.317054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20873
I20260620 18:34:38.317241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:38.317595 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.317806 388 external_mini_cluster.cc:1428] Setting key c60b8a83a3292c0895281cdad644ffb9
I20260620 18:34:38.331852 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8: Bootstrap starting.
I20260620 18:34:38.332264 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.319845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44631 (local address 127.0.0.1:36691)
0620 18:34:38.320794 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.320806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.320849 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:34:38.322698 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.322734 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.323181 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.323597 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.324134 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.325588 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.327475 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.328266 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.328472 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.329794 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.329801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.329816 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.330357 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.330449 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.330487 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.330617 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.330965 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.331150 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":448,"threads_started":1}
I20260620 18:34:38.335618 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.336606 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:38.336947 20940 log.cc:824] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.340369 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8: No bootstrap required, opened a new log
I20260620 18:34:38.349572 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } }
I20260620 18:34:38.350159 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.350308 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f530dc6c0854395aaf9d73a5a09a9f8, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.350658 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.341672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47255 (local address 127.0.0.1:36691)
0620 18:34:38.341862 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.341868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.344115 (+ 2247us) server_negotiation.cc:407] Connection header received
0620 18:34:38.344258 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.344264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.344343 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.344461 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.344944 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.346185 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.347683 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.348397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.348533 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.349753 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.349759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.349764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.349786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.349830 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.349853 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.349970 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.350211 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.350269 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:34:38.350975 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } }
I20260620 18:34:38.351234 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:38.351413 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:38.351655 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.354050 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } }
I20260620 18:34:38.354498 20940 leader_election.cc:302] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f530dc6c0854395aaf9d73a5a09a9f8; no voters:
I20260620 18:34:38.355183 20940 leader_election.cc:288] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:38.355371 20950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:38.356165 20950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7f530dc6c0854395aaf9d73a5a09a9f8, State: Running, Role: LEADER
I20260620 18:34:38.356786 20950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } }
I20260620 18:34:38.357352 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:38.360028 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } } }
I20260620 18:34:38.360046 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f530dc6c0854395aaf9d73a5a09a9f8. Latest consensus state: current_term: 1 leader_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f530dc6c0854395aaf9d73a5a09a9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36691 } } }
I20260620 18:34:38.360641 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.360662 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.362299 20957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:38.366751 20957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:38.374883 20957 catalog_manager.cc:1347] Generated new cluster ID: 57f70aa5539847a4bec02ff87d7951a7
I20260620 18:34:38.375078 20957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:38.387789 20957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:38.389215 20957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:38.402978 20957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f530dc6c0854395aaf9d73a5a09a9f8: Generated new TSK 0
I20260620 18:34:38.404127 20957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:38.418066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20873
W20260620 18:34:38.419277 20935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44631
W20260620 18:34:38.420214 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:38.420424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20873
W20260620 18:34:38.437505 20858 connection.cc:535] client connection to 127.0.0.1:36691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36691): Connection reset by peer (error 104)
2026-06-20T18:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:38 dist-test-slave-mcnm krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:38 dist-test-slave-mcnm krb5kdc[20971](info): set up 2 sockets
Jun 20 18:34:38 dist-test-slave-mcnm krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260620 18:34:39.524065 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:39 dist-test-slave-mcnm krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:39.783385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:40.126396 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:40.126706 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:40.126880 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:40.134418 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:40.134547 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:40.134639 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:40.134739 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:40.134812 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:40.145084 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:40.146541 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:40.148103 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:40.160591 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.160702 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.160723 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:40.161687 20984 server_base.cc:1034] running on GCE node
I20260620 18:34:40.162565 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:40.164135 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:40.165397 20984 hybrid_clock.cc:648] HybridClock initialized: now 1781980480165350 us; error 38 us; skew 500 ppm
I20260620 18:34:40.165773 20984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:40 dist-test-slave-mcnm krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:40.172976 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:40.174535 20984 webserver.cc:460] Webserver started at http://127.0.0.1:41287/ using document root <none> and password file <none>
I20260620 18:34:40.175268 20984 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:40.175426 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:40.175856 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:40.178983 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "50eb7dd8f78644bfb2db074afdef197a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-mcnm"
server_key: "e6465f1caceee83c9c538e579cfab4f2"
server_key_iv: "cb3cfd03111078e0b984bb85d86c75f8"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.179839 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "50eb7dd8f78644bfb2db074afdef197a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-mcnm"
server_key: "e6465f1caceee83c9c538e579cfab4f2"
server_key_iv: "cb3cfd03111078e0b984bb85d86c75f8"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.185079 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:34:40.188621 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:40.190132 20984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:34:40.190325 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "50eb7dd8f78644bfb2db074afdef197a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-mcnm"
server_key: "e6465f1caceee83c9c538e579cfab4f2"
server_key_iv: "cb3cfd03111078e0b984bb85d86c75f8"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.190531 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:40.196460 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:40.200853 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:40.201165 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:40.219916 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37467
I20260620 18:34:40.220003 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37467 every 8 connection(s)
I20260620 18:34:40.221333 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:40.225059 21051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:40.229792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 20984
I20260620 18:34:40.230006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:40.230602 388 external_mini_cluster.cc:1428] Setting key cc6c753686c4c216b679a47db6d09ed8
I20260620 18:34:40.237638 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a: Bootstrap starting.
I20260620 18:34:40.239327 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:40.232454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50891 (local address 127.0.0.1:37467)
0620 18:34:40.233326 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:40.233343 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:40.233400 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:34:40.235843 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:40.235867 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:40.236215 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:40.236548 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:40.237689 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:40.237697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:40.237710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:40.237727 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:40.237786 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:40.237842 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:40.238030 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:40.238167 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:34:40.238367 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":408,"threads_started":1}
I20260620 18:34:40.241029 21051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:40.242228 21051 log.cc:824] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:40.243060 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:40.245520 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a: No bootstrap required, opened a new log
I20260620 18:34:40.251063 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:40.248446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49949 (local address 127.0.0.1:37467)
0620 18:34:40.248662 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:40.248673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:40.248996 (+ 323us) server_negotiation.cc:407] Connection header received
0620 18:34:40.249163 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:40.249167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:40.249221 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:40.249306 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:40.250305 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:40.250309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:40.250312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:40.250328 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:40.250368 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:40.250383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:40.250487 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:40.250655 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:34:40.250710 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:34:40.255420 21051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260620 18:34:40.255903 21051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:40.256024 21051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50eb7dd8f78644bfb2db074afdef197a, State: Initialized, Role: FOLLOWER
I20260620 18:34:40.256629 21051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260620 18:34:40.256865 21051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:40.257019 21051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:40.257211 21051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:40.259666 21051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260620 18:34:40.260236 21051 leader_election.cc:302] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50eb7dd8f78644bfb2db074afdef197a; no voters:
I20260620 18:34:40.261058 21051 leader_election.cc:288] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:40.261191 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:40.262308 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [term 1 LEADER]: Becoming Leader. State: Replica: 50eb7dd8f78644bfb2db074afdef197a, State: Running, Role: LEADER
I20260620 18:34:40.262949 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260620 18:34:40.265106 21051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:40.266705 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50eb7dd8f78644bfb2db074afdef197a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260620 18:34:40.266712 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50eb7dd8f78644bfb2db074afdef197a. Latest consensus state: current_term: 1 leader_uuid: "50eb7dd8f78644bfb2db074afdef197a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb7dd8f78644bfb2db074afdef197a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260620 18:34:40.267449 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:40.267967 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:40.269845 21067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:40.274807 21067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:40.286556 21067 catalog_manager.cc:1347] Generated new cluster ID: 475957a0b4d849eaa229fcb117a04bc8
I20260620 18:34:40.286702 21067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:40.298542 21067 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:40.298974 21067 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:40.304275 21067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50eb7dd8f78644bfb2db074afdef197a: Generated new TSK 0
I20260620 18:34:40.305074 21067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:40.319056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20984
W20260620 18:34:40.320571 21046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50891
W20260620 18:34:40.321266 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 20984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:40.321416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 20984
2026-06-20T18:34:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:40 dist-test-slave-mcnm krb5kdc[21082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:40 dist-test-slave-mcnm krb5kdc[21082](info): set up 2 sockets
Jun 20 18:34:40 dist-test-slave-mcnm krb5kdc[21082](info): commencing operation
krb5kdc: starting...
W20260620 18:34:41.428050 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:41 dist-test-slave-mcnm krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:41.687222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:42.023669 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:42.023958 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:42.024117 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:42.031579 21095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:42.031701 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:42.031817 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:42.031932 21095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:42.031986 21095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:42.042086 21095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:42.043498 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:42.044930 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:42.057188 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.057235 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.057436 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.058626 21095 server_base.cc:1034] running on GCE node
I20260620 18:34:42.059505 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:42.060963 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:42.062181 21095 hybrid_clock.cc:648] HybridClock initialized: now 1781980482062133 us; error 38 us; skew 500 ppm
I20260620 18:34:42.062553 21095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:42 dist-test-slave-mcnm krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:42.065850 21095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:42.067502 21095 webserver.cc:460] Webserver started at http://127.0.0.1:35479/ using document root <none> and password file <none>
I20260620 18:34:42.068269 21095 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:42.068418 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:42.068732 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:42.071834 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-mcnm"
server_key: "ffe4220cad24c8ee589a6339ccec2cff"
server_key_iv: "45d730015f14888ee2d823036123dbbd"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.072610 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-mcnm"
server_key: "ffe4220cad24c8ee589a6339ccec2cff"
server_key_iv: "45d730015f14888ee2d823036123dbbd"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.077944 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:34:42.081640 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.082986 21095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:34:42.083212 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-mcnm"
server_key: "ffe4220cad24c8ee589a6339ccec2cff"
server_key_iv: "45d730015f14888ee2d823036123dbbd"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.083420 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:42.100498 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:42.105052 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:42.105363 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:42.124498 21095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42787
I20260620 18:34:42.124533 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42787 every 8 connection(s)
I20260620 18:34:42.125902 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:42.129786 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:42.132833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21095
I20260620 18:34:42.133054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:42.133512 388 external_mini_cluster.cc:1428] Setting key d5ce0826870ee2c472b04913e6c606d5
I20260620 18:34:42.142128 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9: Bootstrap starting.
I20260620 18:34:42.145484 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:42.146509 21162 log.cc:824] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:34:42 dist-test-slave-mcnm krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:34:42.149818 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9: No bootstrap required, opened a new log
I20260620 18:34:42.156476 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.135409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38721 (local address 127.0.0.1:42787)
0620 18:34:42.136284 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:42.136297 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:42.136343 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:34:42.138070 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:42.138092 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:42.138512 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:42.138927 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:42.140471 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.141761 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.143142 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.143904 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.144135 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.148669 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:42.148677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:42.148693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:42.148731 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:42.152135 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.152930 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.152937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.152944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.153120 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.153520 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.153531 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.153534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.154091 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:42.154477 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:42.154928 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:34:42.155361 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":369,"threads_started":1}
I20260620 18:34:42.158334 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 18:34:42.158794 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:42.158901 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9a0b6e948149b0ad1f88f6ea1b08e9, State: Initialized, Role: FOLLOWER
I20260620 18:34:42.159559 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 18:34:42.159847 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:42.160006 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:42.160183 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:42.160694 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:42.162510 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 18:34:42.163013 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9a0b6e948149b0ad1f88f6ea1b08e9; no voters:
I20260620 18:34:42.163723 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:42.163862 21169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:42.166203 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:42.170222 21169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c9a0b6e948149b0ad1f88f6ea1b08e9, State: Running, Role: LEADER
I20260620 18:34:42.171036 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 18:34:42.174060 21172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260620 18:34:42.174427 21172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:42.175048 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9a0b6e948149b0ad1f88f6ea1b08e9. Latest consensus state: current_term: 1 leader_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9a0b6e948149b0ad1f88f6ea1b08e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260620 18:34:42.175390 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:42.194159 21182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:42.195377 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.165479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60299 (local address 127.0.0.1:42787)
0620 18:34:42.165682 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:42.165688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:42.171198 (+ 5510us) server_negotiation.cc:407] Connection header received
0620 18:34:42.171585 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:42.171594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:42.171716 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:42.171821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:42.174840 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.176506 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.179212 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.180140 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.180295 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.185468 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:42.185480 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:42.185488 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:42.185546 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:42.190568 (+ 5022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.191837 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.191846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.191853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.192082 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.192613 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.192621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.192627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.193260 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:42.193548 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:42.194369 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 18:34:42.194696 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:34:42.201516 21182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:42.210060 21182 catalog_manager.cc:1347] Generated new cluster ID: 9eaeaba17264429a9b03bbb540c0782a
I20260620 18:34:42.210191 21182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:42.244771 21182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:42.246120 21182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:42.255386 21182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c9a0b6e948149b0ad1f88f6ea1b08e9: Generated new TSK 0
I20260620 18:34:42.256209 21182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:42.264855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21095
I20260620 18:34:42.265882 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:42.362699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21095
2026-06-20T18:34:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:42 dist-test-slave-mcnm krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:42 dist-test-slave-mcnm krb5kdc[21195](info): set up 2 sockets
Jun 20 18:34:42 dist-test-slave-mcnm krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260620 18:34:43.488080 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:43 dist-test-slave-mcnm krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:43.745450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:44.083235 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:44.083564 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:44.083745 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:44.091787 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:44.091900 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:44.091989 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:44.092092 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:44.092160 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:44.101982 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:44.103245 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:44.104691 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:44.116649 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:44.116686 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:44.116775 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:44.118398 21208 server_base.cc:1034] running on GCE node
I20260620 18:34:44.119174 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:44.120648 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:44.121878 21208 hybrid_clock.cc:648] HybridClock initialized: now 1781980484121810 us; error 54 us; skew 500 ppm
I20260620 18:34:44.122251 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:44 dist-test-slave-mcnm krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:44.125636 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:44.127183 21208 webserver.cc:460] Webserver started at http://127.0.0.1:41005/ using document root <none> and password file <none>
I20260620 18:34:44.127962 21208 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:44.128108 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:44.128432 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:44.131560 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f"
format_stamp: "Formatted at 2026-06-20 18:34:44 on dist-test-slave-mcnm"
server_key: "24af818a64a0cedae3b733b91de98784"
server_key_iv: "828f5b991a7ee0fd9dc658b12c48138f"
server_key_version: "encryptionkey@0"
I20260620 18:34:44.132380 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f"
format_stamp: "Formatted at 2026-06-20 18:34:44 on dist-test-slave-mcnm"
server_key: "24af818a64a0cedae3b733b91de98784"
server_key_iv: "828f5b991a7ee0fd9dc658b12c48138f"
server_key_version: "encryptionkey@0"
I20260620 18:34:44.137756 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:34:44.141304 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:44.142745 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:34:44.142967 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f"
format_stamp: "Formatted at 2026-06-20 18:34:44 on dist-test-slave-mcnm"
server_key: "24af818a64a0cedae3b733b91de98784"
server_key_iv: "828f5b991a7ee0fd9dc658b12c48138f"
server_key_version: "encryptionkey@0"
I20260620 18:34:44.143251 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:44.160811 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:44.165356 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:44.165650 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:44.185286 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45685
I20260620 18:34:44.185319 21274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45685 every 8 connection(s)
I20260620 18:34:44.186923 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:44.190860 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:44.196166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21208
I20260620 18:34:44.196408 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:44.196943 388 external_mini_cluster.cc:1428] Setting key 0e85aba04e8ae4f0c99d199337c3adae
I20260620 18:34:44.202431 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f: Bootstrap starting.
I20260620 18:34:44.206275 21275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:44.207470 21275 log.cc:824] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:44.211247 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f: No bootstrap required, opened a new log
I20260620 18:34:44.211252 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.199024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49767 (local address 127.0.0.1:45685)
0620 18:34:44.199985 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.200000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.200045 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:34:44.201774 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.201797 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.202232 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.202594 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.203183 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.204601 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.206559 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.207250 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.207382 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.208695 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.208709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.208731 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:44.209444 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.209518 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:44.209543 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.209632 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.209961 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.210238 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":486,"threads_started":1}
I20260620 18:34:44.215104 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:44.220384 21275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } }
I20260620 18:34:44.220818 21275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:44.220955 21275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0def7f03bf4ca39ea6c761fdc1b78f, State: Initialized, Role: FOLLOWER
I20260620 18:34:44.221664 21275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } }
I20260620 18:34:44.221952 21275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:44.222128 21275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:44.222332 21275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:44.224872 21275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } }
I20260620 18:34:44.225445 21275 leader_election.cc:302] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e0def7f03bf4ca39ea6c761fdc1b78f; no voters:
I20260620 18:34:44.225854 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.219492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52107 (local address 127.0.0.1:45685)
0620 18:34:44.219713 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.219721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.220116 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:34:44.220332 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.220336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.220404 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.220505 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.221097 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.221818 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.223300 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.223852 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.224005 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.225092 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.225097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.225101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:44.225122 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.225169 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:44.225191 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.225283 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.225481 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.225569 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:34:44.226567 21275 leader_election.cc:288] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:44.226609 21285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:44.228790 21275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:44.229213 21285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [term 1 LEADER]: Becoming Leader. State: Replica: 9e0def7f03bf4ca39ea6c761fdc1b78f, State: Running, Role: LEADER
I20260620 18:34:44.229749 21285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } }
I20260620 18:34:44.232803 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } } }
I20260620 18:34:44.233197 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.232779 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e0def7f03bf4ca39ea6c761fdc1b78f. Latest consensus state: current_term: 1 leader_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0def7f03bf4ca39ea6c761fdc1b78f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45685 } } }
I20260620 18:34:44.234588 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.240206 21294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:44.246102 21294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:44.255577 21294 catalog_manager.cc:1347] Generated new cluster ID: cdb19e6040ff490a961c7097ddf75388
I20260620 18:34:44.255722 21294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:44.274709 21294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:44.276358 21294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:44.287019 21294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e0def7f03bf4ca39ea6c761fdc1b78f: Generated new TSK 0
I20260620 18:34:44.288118 21294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:44.304793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21208
W20260620 18:34:44.305925 21270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49767
W20260620 18:34:44.306619 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:44.306783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21208
2026-06-20T18:34:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:44 dist-test-slave-mcnm krb5kdc[21306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:44 dist-test-slave-mcnm krb5kdc[21306](info): set up 2 sockets
Jun 20 18:34:44 dist-test-slave-mcnm krb5kdc[21306](info): commencing operation
krb5kdc: starting...
W20260620 18:34:45.420056 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:45 dist-test-slave-mcnm krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:45.675285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:46.013867 21319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:46.014241 21319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:46.014407 21319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:46.022008 21319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:46.022172 21319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:46.022274 21319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:46.022383 21319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:46.022459 21319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:46.032449 21319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:46.033824 21319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:46.035161 21319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:46.047627 21324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:46.047550 21325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:46.047551 21327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:46.049093 21319 server_base.cc:1034] running on GCE node
I20260620 18:34:46.049871 21319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:46.051342 21319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:46.052634 21319 hybrid_clock.cc:648] HybridClock initialized: now 1781980486052580 us; error 48 us; skew 500 ppm
I20260620 18:34:46.053028 21319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:46 dist-test-slave-mcnm krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:46.061075 21319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:46.062594 21319 webserver.cc:460] Webserver started at http://127.0.0.1:44073/ using document root <none> and password file <none>
I20260620 18:34:46.063247 21319 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:46.063346 21319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:46.063684 21319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:46.066584 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "2764a13f2172415ab47ba48b3d1d1864"
format_stamp: "Formatted at 2026-06-20 18:34:46 on dist-test-slave-mcnm"
server_key: "29e8bbbf058911c849be383d691c7b0f"
server_key_iv: "45534b9b38e1aa75a6d40d7ce461260d"
server_key_version: "encryptionkey@0"
I20260620 18:34:46.067278 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "2764a13f2172415ab47ba48b3d1d1864"
format_stamp: "Formatted at 2026-06-20 18:34:46 on dist-test-slave-mcnm"
server_key: "29e8bbbf058911c849be383d691c7b0f"
server_key_iv: "45534b9b38e1aa75a6d40d7ce461260d"
server_key_version: "encryptionkey@0"
I20260620 18:34:46.072276 21319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:34:46.075580 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:46.076898 21319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:34:46.077077 21319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "2764a13f2172415ab47ba48b3d1d1864"
format_stamp: "Formatted at 2026-06-20 18:34:46 on dist-test-slave-mcnm"
server_key: "29e8bbbf058911c849be383d691c7b0f"
server_key_iv: "45534b9b38e1aa75a6d40d7ce461260d"
server_key_version: "encryptionkey@0"
I20260620 18:34:46.077272 21319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:46.082930 21319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:46.087054 21319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:46.087388 21319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:46.106135 21319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45089
I20260620 18:34:46.106220 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45089 every 8 connection(s)
I20260620 18:34:46.107550 21319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:46.110828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21319
I20260620 18:34:46.111037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:46.111582 388 external_mini_cluster.cc:1428] Setting key 03c291952fa33be26394121743365125
I20260620 18:34:46.111742 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:46.119778 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:46.113365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43783 (local address 127.0.0.1:45089)
0620 18:34:46.114142 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:46.114157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:46.114197 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:34:46.116325 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:46.116347 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:46.116725 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:46.117063 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:46.118104 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:46.118111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:46.118127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:46.118140 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:46.118198 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:46.118220 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:46.118399 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:46.118537 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:34:46.118756 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":353,"threads_started":1}
I20260620 18:34:46.123479 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:46.125710 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864: Bootstrap starting.
I20260620 18:34:46.129738 21386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:46.131021 21386 log.cc:824] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:46.131481 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:46.128599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35763 (local address 127.0.0.1:45089)
0620 18:34:46.128789 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:46.128795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:46.129379 (+ 584us) server_negotiation.cc:407] Connection header received
0620 18:34:46.129531 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:46.129539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:46.129621 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:46.129747 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:46.130810 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:46.130817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:46.130822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:46.130841 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:46.130888 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:46.130908 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:46.131039 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:46.131145 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:34:46.131203 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:34:46.134052 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864: No bootstrap required, opened a new log
I20260620 18:34:46.141706 21386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260620 18:34:46.142098 21386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:46.142220 21386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2764a13f2172415ab47ba48b3d1d1864, State: Initialized, Role: FOLLOWER
I20260620 18:34:46.142784 21386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260620 18:34:46.143069 21386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:46.143222 21386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:46.143392 21386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:46.145819 21386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260620 18:34:46.146369 21386 leader_election.cc:302] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2764a13f2172415ab47ba48b3d1d1864; no voters:
I20260620 18:34:46.146996 21386 leader_election.cc:288] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:46.147159 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:46.147956 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [term 1 LEADER]: Becoming Leader. State: Replica: 2764a13f2172415ab47ba48b3d1d1864, State: Running, Role: LEADER
I20260620 18:34:46.148546 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260620 18:34:46.149246 21386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:46.151746 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2764a13f2172415ab47ba48b3d1d1864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260620 18:34:46.151584 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2764a13f2172415ab47ba48b3d1d1864. Latest consensus state: current_term: 1 leader_uuid: "2764a13f2172415ab47ba48b3d1d1864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2764a13f2172415ab47ba48b3d1d1864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260620 18:34:46.152174 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:46.152217 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:46.154336 21403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:46.159667 21403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:46.168876 21403 catalog_manager.cc:1347] Generated new cluster ID: cab7cfc49bff4f77868daf04697da01b
I20260620 18:34:46.169010 21403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:46.205309 21403 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:46.205762 21403 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:46.212206 21403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2764a13f2172415ab47ba48b3d1d1864: Generated new TSK 0
I20260620 18:34:46.212999 21403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:46.227133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21319
W20260620 18:34:46.228381 21381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43783
W20260620 18:34:46.229076 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:46.229249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21319
2026-06-20T18:34:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:46 dist-test-slave-mcnm krb5kdc[21417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:46 dist-test-slave-mcnm krb5kdc[21417](info): set up 2 sockets
Jun 20 18:34:46 dist-test-slave-mcnm krb5kdc[21417](info): commencing operation
krb5kdc: starting...
W20260620 18:34:47.351931 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:47 dist-test-slave-mcnm krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:47.602281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:47.929744 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:47.930097 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:47.930284 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:47.937523 21430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:47.937674 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:47.937788 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:47.937906 21430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:47.937994 21430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:47.947630 21430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:47.949036 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:47.950361 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:47.962631 21436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:47.962699 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:47.962721 21435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:47.963430 21430 server_base.cc:1034] running on GCE node
I20260620 18:34:47.964290 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:47.965796 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:47.967029 21430 hybrid_clock.cc:648] HybridClock initialized: now 1781980487966986 us; error 68 us; skew 500 ppm
I20260620 18:34:47.967408 21430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:47 dist-test-slave-mcnm krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:47.970598 21430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:47.972158 21430 webserver.cc:460] Webserver started at http://127.0.0.1:45709/ using document root <none> and password file <none>
I20260620 18:34:47.972836 21430 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:47.972986 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:47.973284 21430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:47.976271 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "0b668a1504014392b4dac0786a28783b"
format_stamp: "Formatted at 2026-06-20 18:34:47 on dist-test-slave-mcnm"
server_key: "5cb3d03d11d193bc491f0f7599769bc0"
server_key_iv: "e22eaab4e8c763abf97d3d76ede45a50"
server_key_version: "encryptionkey@0"
I20260620 18:34:47.977033 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b668a1504014392b4dac0786a28783b"
format_stamp: "Formatted at 2026-06-20 18:34:47 on dist-test-slave-mcnm"
server_key: "5cb3d03d11d193bc491f0f7599769bc0"
server_key_iv: "e22eaab4e8c763abf97d3d76ede45a50"
server_key_version: "encryptionkey@0"
I20260620 18:34:47.982100 21430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:34:47.985581 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:47.986938 21430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:34:47.987128 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "0b668a1504014392b4dac0786a28783b"
format_stamp: "Formatted at 2026-06-20 18:34:47 on dist-test-slave-mcnm"
server_key: "5cb3d03d11d193bc491f0f7599769bc0"
server_key_iv: "e22eaab4e8c763abf97d3d76ede45a50"
server_key_version: "encryptionkey@0"
I20260620 18:34:47.987321 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:48.010766 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:48.015082 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:48.015319 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:48.032884 21430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38267
I20260620 18:34:48.032917 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38267 every 8 connection(s)
I20260620 18:34:48.034322 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:48.036671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21430
I20260620 18:34:48.036880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:48.037453 388 external_mini_cluster.cc:1428] Setting key 7699fa173bfbb9966335255fb35cb1ea
I20260620 18:34:48.038880 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:34:48 dist-test-slave-mcnm krb5kdc[21417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:34:48.053114 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b: Bootstrap starting.
I20260620 18:34:48.056427 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:48.057497 21497 log.cc:824] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:48.060485 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b: No bootstrap required, opened a new log
I20260620 18:34:48.061002 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.039302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:39375 (local address 127.0.0.1:38267)
0620 18:34:48.040088 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:48.040100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:48.040141 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:34:48.041771 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:48.041792 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:48.042212 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:48.042553 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:48.044452 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.045831 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.047649 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.048287 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.048425 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.053406 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:48.053419 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:48.053434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:48.053474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:48.056701 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.057560 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.057566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.057572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.057777 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.058245 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.058250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.058257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.058799 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:48.059137 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:48.059662 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 18:34:48.059982 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260620 18:34:48.064859 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:48.068720 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } }
I20260620 18:34:48.069178 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:48.069278 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b668a1504014392b4dac0786a28783b, State: Initialized, Role: FOLLOWER
I20260620 18:34:48.069974 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } }
I20260620 18:34:48.070247 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:48.070417 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:48.070591 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:48.073447 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } }
I20260620 18:34:48.073988 21497 leader_election.cc:302] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b668a1504014392b4dac0786a28783b; no voters:
I20260620 18:34:48.074659 21497 leader_election.cc:288] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:48.074781 21507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:48.076817 21507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [term 1 LEADER]: Becoming Leader. State: Replica: 0b668a1504014392b4dac0786a28783b, State: Running, Role: LEADER
I20260620 18:34:48.077126 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:48.077486 21507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } }
I20260620 18:34:48.080240 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b668a1504014392b4dac0786a28783b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } } }
I20260620 18:34:48.080621 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:48.083537 21512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:48.083446 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b668a1504014392b4dac0786a28783b. Latest consensus state: current_term: 1 leader_uuid: "0b668a1504014392b4dac0786a28783b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b668a1504014392b4dac0786a28783b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38267 } } }
I20260620 18:34:48.083832 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:48.086619 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.069027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:44925 (local address 127.0.0.1:38267)
0620 18:34:48.069221 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:48.069228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:48.069707 (+ 479us) server_negotiation.cc:407] Connection header received
0620 18:34:48.069880 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:48.069886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:48.069972 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:48.070082 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:48.071607 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.072753 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.074082 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.074852 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.075012 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.078518 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:48.078524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:48.078528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:48.078563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:48.082818 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.083974 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.083982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.083988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.084215 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.084916 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.084923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.084929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.085336 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:48.085677 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:48.086003 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:34:48.086186 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:34:48.091470 21512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:48.101755 21512 catalog_manager.cc:1347] Generated new cluster ID: 37d7ca90ea2644439431c546904efdd1
I20260620 18:34:48.101863 21512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:48.113142 21512 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:48.114694 21512 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:48.121660 21512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b668a1504014392b4dac0786a28783b: Generated new TSK 0
I20260620 18:34:48.122548 21512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:48.127916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21430
I20260620 18:34:48.129655 21492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:48.220273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21430
W20260620 18:34:48.237237 21415 connection.cc:535] client connection to 127.0.0.1:38267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38267): Connection reset by peer (error 104)
2026-06-20T18:34:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:48 dist-test-slave-mcnm krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:48 dist-test-slave-mcnm krb5kdc[21529](info): set up 2 sockets
Jun 20 18:34:48 dist-test-slave-mcnm krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260620 18:34:49.328011 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:49 dist-test-slave-mcnm krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:49.584125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:49.919122 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:49.919417 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:49.919564 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:49.926779 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:49.926898 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:49.926946 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:49.927001 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:49.927047 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:49.937197 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:49.938509 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:49.939781 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:49.951928 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.952054 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.952021 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:49.953161 21542 server_base.cc:1034] running on GCE node
I20260620 18:34:49.954111 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:49.955629 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:49.956884 21542 hybrid_clock.cc:648] HybridClock initialized: now 1781980489956798 us; error 72 us; skew 500 ppm
I20260620 18:34:49.957288 21542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:49 dist-test-slave-mcnm krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:49.960459 21542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:49.962008 21542 webserver.cc:460] Webserver started at http://127.0.0.1:38001/ using document root <none> and password file <none>
I20260620 18:34:49.962697 21542 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:49.962844 21542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:49.963171 21542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:49.966195 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "d07163f759f04b85845db4de5a85e162"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-mcnm"
server_key: "833aace9488f60f8d8b5340a560f7fb3"
server_key_iv: "44f8224e84f4f7bcd29712e038325324"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.967005 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "d07163f759f04b85845db4de5a85e162"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-mcnm"
server_key: "833aace9488f60f8d8b5340a560f7fb3"
server_key_iv: "44f8224e84f4f7bcd29712e038325324"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.972571 21542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:34:49.976074 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.977581 21542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:34:49.977775 21542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "d07163f759f04b85845db4de5a85e162"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-mcnm"
server_key: "833aace9488f60f8d8b5340a560f7fb3"
server_key_iv: "44f8224e84f4f7bcd29712e038325324"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.977983 21542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:50.003393 21542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:50.008224 21542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:50.008487 21542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:50.026736 21542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35027
I20260620 18:34:50.026758 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35027 every 8 connection(s)
I20260620 18:34:50.028204 21542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:50.030046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21542
I20260620 18:34:50.030256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:50.030737 388 external_mini_cluster.cc:1428] Setting key a91086c362a54ad2f29f1e207c255599
I20260620 18:34:50.032809 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:50.046576 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:50.032692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:34075 (local address 127.0.0.1:35027)
0620 18:34:50.033502 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:50.033519 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:50.033568 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:34:50.035881 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:50.035910 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:50.036516 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:50.036958 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:50.037492 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.039561 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.040957 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.041977 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.042162 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:50.043708 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:50.043718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:50.043742 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:50.044510 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:50.044627 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:50.044665 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:50.044837 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:50.045140 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:34:50.045321 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":330,"threads_started":1}
I20260620 18:34:50.047283 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162: Bootstrap starting.
I20260620 18:34:50.050472 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:50.050806 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:50.051833 21609 log.cc:824] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:50.055248 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162: No bootstrap required, opened a new log
I20260620 18:34:50.062057 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:50.055032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:57659 (local address 127.0.0.1:35027)
0620 18:34:50.055218 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:50.055226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:50.055677 (+ 451us) server_negotiation.cc:407] Connection header received
0620 18:34:50.055854 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:50.055859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:50.055940 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:50.056052 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:50.056598 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.057765 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.059167 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.059877 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.060013 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:50.061211 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:50.061218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:50.061223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:50.061243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:50.061292 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:50.061313 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:50.061443 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:50.061669 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:34:50.061728 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:34:50.065044 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 18:34:50.065393 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:50.065465 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07163f759f04b85845db4de5a85e162, State: Initialized, Role: FOLLOWER
I20260620 18:34:50.065943 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 18:34:50.066139 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:50.066241 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:50.066360 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:50.068578 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 18:34:50.069021 21609 leader_election.cc:302] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d07163f759f04b85845db4de5a85e162; no voters:
I20260620 18:34:50.069736 21609 leader_election.cc:288] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:50.069875 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:50.070927 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [term 1 LEADER]: Becoming Leader. State: Replica: d07163f759f04b85845db4de5a85e162, State: Running, Role: LEADER
I20260620 18:34:50.071437 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 18:34:50.071933 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:50.074035 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d07163f759f04b85845db4de5a85e162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260620 18:34:50.074544 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:50.074616 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d07163f759f04b85845db4de5a85e162. Latest consensus state: current_term: 1 leader_uuid: "d07163f759f04b85845db4de5a85e162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07163f759f04b85845db4de5a85e162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260620 18:34:50.074895 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:50.077682 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:50.083389 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:50.092661 21626 catalog_manager.cc:1347] Generated new cluster ID: 1002c9dc21424dc094a0f05daf2e773c
I20260620 18:34:50.092782 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:50.102991 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:50.105094 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:50.115041 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d07163f759f04b85845db4de5a85e162: Generated new TSK 0
I20260620 18:34:50.116179 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:50.132450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21542
W20260620 18:34:50.133903 21604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:34075
W20260620 18:34:50.134732 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:50.134963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21542
W20260620 18:34:50.151999 21527 connection.cc:535] client connection to 127.0.0.1:35027 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35027): Connection reset by peer (error 104)
2026-06-20T18:34:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:50 dist-test-slave-mcnm krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:50 dist-test-slave-mcnm krb5kdc[21640](info): set up 2 sockets
Jun 20 18:34:50 dist-test-slave-mcnm krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260620 18:34:51.255939 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:51 dist-test-slave-mcnm krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:51.512245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:51.840330 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:51.840644 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:51.840736 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:51.847975 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:51.848085 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:51.848132 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:51.848194 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:51.848232 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:51.858419 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:51.860056 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:51.861474 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:51.873816 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:51.873848 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:51.873883 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:51.874708 21653 server_base.cc:1034] running on GCE node
I20260620 18:34:51.875452 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:51.876952 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:51.878239 21653 hybrid_clock.cc:648] HybridClock initialized: now 1781980491878194 us; error 42 us; skew 500 ppm
I20260620 18:34:51.878623 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:51 dist-test-slave-mcnm krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:51.887976 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:51.889714 21653 webserver.cc:460] Webserver started at http://127.0.0.1:36563/ using document root <none> and password file <none>
I20260620 18:34:51.890470 21653 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:51.890627 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:51.890952 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:51.894112 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "8ac611704be74b1183df19bd49e5ab93"
format_stamp: "Formatted at 2026-06-20 18:34:51 on dist-test-slave-mcnm"
server_key: "887ff196c86c6521213a30524be10aa3"
server_key_iv: "0cd592f685a08d9700fa7c4f39aee77d"
server_key_version: "encryptionkey@0"
I20260620 18:34:51.894893 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ac611704be74b1183df19bd49e5ab93"
format_stamp: "Formatted at 2026-06-20 18:34:51 on dist-test-slave-mcnm"
server_key: "887ff196c86c6521213a30524be10aa3"
server_key_iv: "0cd592f685a08d9700fa7c4f39aee77d"
server_key_version: "encryptionkey@0"
I20260620 18:34:51.900180 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 18:34:51.903762 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:51.905102 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:51.905299 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "8ac611704be74b1183df19bd49e5ab93"
format_stamp: "Formatted at 2026-06-20 18:34:51 on dist-test-slave-mcnm"
server_key: "887ff196c86c6521213a30524be10aa3"
server_key_iv: "0cd592f685a08d9700fa7c4f39aee77d"
server_key_version: "encryptionkey@0"
I20260620 18:34:51.905500 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:51.911255 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:51.915685 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:51.915993 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:51.934785 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45337
I20260620 18:34:51.934825 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45337 every 8 connection(s)
I20260620 18:34:51.936237 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:51.936664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21653
I20260620 18:34:51.936867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:51.937477 388 external_mini_cluster.cc:1428] Setting key a255dbbce2464f0b0b101a7861cb2089
I20260620 18:34:51.941181 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:51.949862 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:51.939413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:55543 (local address 127.0.0.1:45337)
0620 18:34:51.944017 (+ 4604us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:51.944029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:51.944072 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:34:51.946309 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:51.946332 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:51.946689 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:51.947133 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:51.948258 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:51.948264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:51.948287 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:51.948304 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:51.948362 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:51.948382 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:51.948638 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:51.948776 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:34:51.948966 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4385,"thread_start_us":4156,"threads_started":1}
I20260620 18:34:51.953614 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:51.954682 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93: Bootstrap starting.
I20260620 18:34:51.958112 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:51.959311 21720 log.cc:824] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:51.960664 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:51.958101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:34869 (local address 127.0.0.1:45337)
0620 18:34:51.958331 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:51.958340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:51.958630 (+ 290us) server_negotiation.cc:407] Connection header received
0620 18:34:51.958821 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:51.958826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:51.958883 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:51.958985 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:51.959964 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:51.959970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:51.959974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:51.959992 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:51.960042 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:51.960058 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:51.960184 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:51.960313 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:34:51.960382 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:34:51.962409 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93: No bootstrap required, opened a new log
I20260620 18:34:51.970036 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260620 18:34:51.970441 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:51.970549 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac611704be74b1183df19bd49e5ab93, State: Initialized, Role: FOLLOWER
I20260620 18:34:51.971135 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260620 18:34:51.971382 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:51.971561 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:51.971737 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:51.974009 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260620 18:34:51.974525 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac611704be74b1183df19bd49e5ab93; no voters:
I20260620 18:34:51.975082 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:51.975239 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:51.975906 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac611704be74b1183df19bd49e5ab93, State: Running, Role: LEADER
I20260620 18:34:51.976460 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260620 18:34:51.977137 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:51.979568 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac611704be74b1183df19bd49e5ab93. Latest consensus state: current_term: 1 leader_uuid: "8ac611704be74b1183df19bd49e5ab93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260620 18:34:51.980424 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:51.979575 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac611704be74b1183df19bd49e5ab93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac611704be74b1183df19bd49e5ab93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260620 18:34:51.980631 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:51.982226 21741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:51.987082 21741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:51.997272 21741 catalog_manager.cc:1347] Generated new cluster ID: 32dc24f39a8742888a0a9e21cd1c242a
I20260620 18:34:51.997394 21741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:52.012809 21741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:52.013235 21741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:52.026949 21741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac611704be74b1183df19bd49e5ab93: Generated new TSK 0
I20260620 18:34:52.027969 21741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:52.045390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21653
W20260620 18:34:52.046612 21715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:55543
W20260620 18:34:52.047309 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:52.047458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21653
2026-06-20T18:34:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:52 dist-test-slave-mcnm krb5kdc[21751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:52 dist-test-slave-mcnm krb5kdc[21751](info): set up 2 sockets
Jun 20 18:34:52 dist-test-slave-mcnm krb5kdc[21751](info): commencing operation
krb5kdc: starting...
W20260620 18:34:53.168057 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:53 dist-test-slave-mcnm krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:53.438910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:53.762610 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.762900 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.762995 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.770299 21764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:53.770435 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.770483 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.770537 21764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:53.770582 21764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:53.780298 21764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:53.781502 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.782709 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.795151 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.795117 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.795077 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.796010 21764 server_base.cc:1034] running on GCE node
I20260620 18:34:53.796746 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.798187 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.799499 21764 hybrid_clock.cc:648] HybridClock initialized: now 1781980493799366 us; error 60 us; skew 500 ppm
I20260620 18:34:53.799928 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:53 dist-test-slave-mcnm krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.803460 21764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:53.805023 21764 webserver.cc:460] Webserver started at http://127.0.0.1:35051/ using document root <none> and password file <none>
I20260620 18:34:53.805704 21764 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.805845 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.806165 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:53.809209 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "591406da45024d6badbdf89af94e33ea"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-mcnm"
server_key: "d7926cb5e3e4b2f9f86073e272c90a98"
server_key_iv: "521395fe1b9cbaaf4bffdb05740302a3"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.810073 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "591406da45024d6badbdf89af94e33ea"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-mcnm"
server_key: "d7926cb5e3e4b2f9f86073e272c90a98"
server_key_iv: "521395fe1b9cbaaf4bffdb05740302a3"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.815212 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:34:53.818682 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.820289 21764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:53.820478 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "591406da45024d6badbdf89af94e33ea"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-mcnm"
server_key: "d7926cb5e3e4b2f9f86073e272c90a98"
server_key_iv: "521395fe1b9cbaaf4bffdb05740302a3"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.820683 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.837234 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.841409 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.841718 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.860559 21764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41915
I20260620 18:34:53.860579 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41915 every 8 connection(s)
I20260620 18:34:53.861948 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:53.862630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21764
I20260620 18:34:53.862836 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:53.863409 388 external_mini_cluster.cc:1428] Setting key fdb8469fc9ce98d3d24a59c858e320b2
I20260620 18:34:53.866184 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:53.876796 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea: Bootstrap starting.
Jun 20 18:34:53 dist-test-slave-mcnm krb5kdc[21751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:34:53.880167 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:53.881191 21831 log.cc:824] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:53.883957 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea: No bootstrap required, opened a new log
I20260620 18:34:53.885553 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.865354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:41475 (local address 127.0.0.1:41915)
0620 18:34:53.866131 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.866144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.866189 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:34:53.867962 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.867983 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.868390 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.868809 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.870250 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.871696 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.872954 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.873633 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.873857 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.878170 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:53.878178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:53.878195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:53.878241 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:53.881502 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.882278 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.882287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.882295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.882474 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.882937 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.882943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.882946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.883435 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:53.883784 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:53.884262 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 18:34:53.884606 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":351,"threads_started":1}
I20260620 18:34:53.889955 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:53.892570 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260620 18:34:53.892970 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:53.893097 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591406da45024d6badbdf89af94e33ea, State: Initialized, Role: FOLLOWER
I20260620 18:34:53.893724 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260620 18:34:53.893960 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:53.894152 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:53.894389 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:53.896919 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260620 18:34:53.897444 21831 leader_election.cc:302] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591406da45024d6badbdf89af94e33ea; no voters:
I20260620 18:34:53.898116 21831 leader_election.cc:288] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:53.898226 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:53.899749 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [term 1 LEADER]: Becoming Leader. State: Replica: 591406da45024d6badbdf89af94e33ea, State: Running, Role: LEADER
I20260620 18:34:53.900409 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260620 18:34:53.900707 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:53.910190 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.894618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:55093 (local address 127.0.0.1:41915)
0620 18:34:53.894839 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.894848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.895249 (+ 401us) server_negotiation.cc:407] Connection header received
0620 18:34:53.895473 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.895478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.895607 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.895726 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.897200 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.898094 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.899970 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.900528 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.900604 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.903924 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:53.903929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:53.903932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:53.903962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:53.906889 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.907651 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.907656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.907660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.908033 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.908578 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.908581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.908584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.908885 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:53.909212 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:53.909608 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:34:53.909782 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:34:53.911399 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591406da45024d6badbdf89af94e33ea. Latest consensus state: current_term: 1 leader_uuid: "591406da45024d6badbdf89af94e33ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260620 18:34:53.911903 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:53.916917 21852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:53.916482 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591406da45024d6badbdf89af94e33ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591406da45024d6badbdf89af94e33ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260620 18:34:53.917098 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:53.923615 21852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:53.933199 21852 catalog_manager.cc:1347] Generated new cluster ID: fe3db747867f472a81de5e93564cabfe
I20260620 18:34:53.933320 21852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:53.954892 21852 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:53.956260 21852 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:53.970446 21852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591406da45024d6badbdf89af94e33ea: Generated new TSK 0
I20260620 18:34:53.971365 21852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:53.976666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21764
I20260620 18:34:53.977669 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:54.069314 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21764
W20260620 18:34:54.086365 21749 connection.cc:535] client connection to 127.0.0.1:41915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41915): Connection reset by peer (error 104)
2026-06-20T18:34:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:54 dist-test-slave-mcnm krb5kdc[21863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:54 dist-test-slave-mcnm krb5kdc[21863](info): set up 2 sockets
Jun 20 18:34:54 dist-test-slave-mcnm krb5kdc[21863](info): commencing operation
krb5kdc: starting...
W20260620 18:34:55.184088 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:55 dist-test-slave-mcnm krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:55.447247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:55.770742 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:55.771091 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:55.771274 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:55.778595 21876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:55.778712 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:55.778810 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:55.778896 21876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:55.778983 21876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:55.788905 21876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:55.790218 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:55.791579 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:55.803758 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:55.803826 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:55.804076 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:55.804826 21876 server_base.cc:1034] running on GCE node
I20260620 18:34:55.805714 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:55.807152 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:55.808352 21876 hybrid_clock.cc:648] HybridClock initialized: now 1781980495808289 us; error 51 us; skew 500 ppm
I20260620 18:34:55.808710 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:55 dist-test-slave-mcnm krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:55.812177 21876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:55.813757 21876 webserver.cc:460] Webserver started at http://127.0.0.1:41383/ using document root <none> and password file <none>
I20260620 18:34:55.814494 21876 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:55.814625 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:55.814990 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:55.817976 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "ba0818f91f944d039a1d03ce75aac92c"
format_stamp: "Formatted at 2026-06-20 18:34:55 on dist-test-slave-mcnm"
server_key: "e945850e9da982f65ebf07f88cacc686"
server_key_iv: "4c7751f90d0c1b5c9ee2d41fdee9618a"
server_key_version: "encryptionkey@0"
I20260620 18:34:55.818704 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba0818f91f944d039a1d03ce75aac92c"
format_stamp: "Formatted at 2026-06-20 18:34:55 on dist-test-slave-mcnm"
server_key: "e945850e9da982f65ebf07f88cacc686"
server_key_iv: "4c7751f90d0c1b5c9ee2d41fdee9618a"
server_key_version: "encryptionkey@0"
I20260620 18:34:55.823900 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:34:55.827322 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:55.828701 21876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:55.828887 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "ba0818f91f944d039a1d03ce75aac92c"
format_stamp: "Formatted at 2026-06-20 18:34:55 on dist-test-slave-mcnm"
server_key: "e945850e9da982f65ebf07f88cacc686"
server_key_iv: "4c7751f90d0c1b5c9ee2d41fdee9618a"
server_key_version: "encryptionkey@0"
I20260620 18:34:55.829083 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:55.858657 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:55.862974 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:55.863278 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:55.882691 21876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43157
I20260620 18:34:55.882721 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43157 every 8 connection(s)
I20260620 18:34:55.884132 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:55.888111 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:55.891607 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21876
I20260620 18:34:55.891790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:55.892351 388 external_mini_cluster.cc:1428] Setting key c36faf24b783a8dc74952dd2a686ecac
I20260620 18:34:55.902017 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c: Bootstrap starting.
I20260620 18:34:55.906009 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:55.907358 21943 log.cc:824] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:55.908196 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:55.894206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:40977 (local address 127.0.0.1:43157)
0620 18:34:55.894945 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:55.894960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:55.895007 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:34:55.897461 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:55.897491 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:55.898061 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:55.898509 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:55.899043 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.901089 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.902723 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.903818 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.904014 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.905205 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:55.905215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:55.905233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:55.905924 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:55.906025 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:55.906073 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:55.906312 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:55.906632 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 18:34:55.906801 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260620 18:34:55.910938 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c: No bootstrap required, opened a new log
I20260620 18:34:55.912599 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:55.919497 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260620 18:34:55.920074 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:55.920186 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0818f91f944d039a1d03ce75aac92c, State: Initialized, Role: FOLLOWER
I20260620 18:34:55.920908 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260620 18:34:55.921211 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:55.921350 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:55.921517 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:55.924407 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:55.917286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:44655 (local address 127.0.0.1:43157)
0620 18:34:55.917499 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:55.917505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:55.917801 (+ 296us) server_negotiation.cc:407] Connection header received
0620 18:34:55.918045 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:55.918049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:55.918122 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:55.918218 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:55.918804 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.919638 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.921570 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.922051 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.922165 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.923510 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:55.923573 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:55.923578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:55.923607 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:55.923672 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:55.923690 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:55.923782 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:55.924007 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:34:55.924050 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:34:55.924758 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260620 18:34:55.925287 21943 leader_election.cc:302] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0818f91f944d039a1d03ce75aac92c; no voters:
I20260620 18:34:55.925894 21943 leader_election.cc:288] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:55.925998 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:55.926892 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [term 1 LEADER]: Becoming Leader. State: Replica: ba0818f91f944d039a1d03ce75aac92c, State: Running, Role: LEADER
I20260620 18:34:55.927455 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260620 18:34:55.928236 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:55.930897 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0818f91f944d039a1d03ce75aac92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260620 18:34:55.931347 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:55.931115 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0818f91f944d039a1d03ce75aac92c. Latest consensus state: current_term: 1 leader_uuid: "ba0818f91f944d039a1d03ce75aac92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0818f91f944d039a1d03ce75aac92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260620 18:34:55.931505 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:55.937101 21958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:55.942049 21958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:55.950444 21958 catalog_manager.cc:1347] Generated new cluster ID: 60319dc648cb44b9962ae17d03e62e4b
I20260620 18:34:55.950588 21958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:55.974635 21958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:55.976413 21958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:55.990634 21958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0818f91f944d039a1d03ce75aac92c: Generated new TSK 0
I20260620 18:34:55.991736 21958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:56.004885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21876
W20260620 18:34:56.005985 21938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:40977
W20260620 18:34:56.006723 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:56.006880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21876
W20260620 18:34:56.024454 21861 connection.cc:535] client connection to 127.0.0.1:43157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43157): Connection reset by peer (error 104)
2026-06-20T18:34:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:56 dist-test-slave-mcnm krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:56 dist-test-slave-mcnm krb5kdc[21974](info): set up 2 sockets
Jun 20 18:34:56 dist-test-slave-mcnm krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260620 18:34:57.116080 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:57 dist-test-slave-mcnm krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:34:57.374552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:57.695793 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:57.696118 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:57.696285 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:57.703312 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:57.703473 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:57.703600 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:57.703722 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:57.703811 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:57.713636 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:57.714973 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:57.716331 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:57.728399 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:57.728448 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:57.728960 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:57.729225 21987 server_base.cc:1034] running on GCE node
I20260620 18:34:57.729974 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:57.731325 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:57.732561 21987 hybrid_clock.cc:648] HybridClock initialized: now 1781980497732515 us; error 40 us; skew 500 ppm
I20260620 18:34:57.732901 21987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:57 dist-test-slave-mcnm krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:57.740337 21987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:34:57.741811 21987 webserver.cc:460] Webserver started at http://127.0.0.1:44779/ using document root <none> and password file <none>
I20260620 18:34:57.742506 21987 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:57.742655 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:57.742949 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:57.745965 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "943c4595c7074bfcab02395f9e9a3d8d"
format_stamp: "Formatted at 2026-06-20 18:34:57 on dist-test-slave-mcnm"
server_key: "ba384516d6e9d10bf5342b5fcc98d734"
server_key_iv: "43ccce4e0cd592d49b4a2ecd0c6635b1"
server_key_version: "encryptionkey@0"
I20260620 18:34:57.746709 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "943c4595c7074bfcab02395f9e9a3d8d"
format_stamp: "Formatted at 2026-06-20 18:34:57 on dist-test-slave-mcnm"
server_key: "ba384516d6e9d10bf5342b5fcc98d734"
server_key_iv: "43ccce4e0cd592d49b4a2ecd0c6635b1"
server_key_version: "encryptionkey@0"
I20260620 18:34:57.751847 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:34:57.755250 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:57.756745 21987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:57.756935 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "943c4595c7074bfcab02395f9e9a3d8d"
format_stamp: "Formatted at 2026-06-20 18:34:57 on dist-test-slave-mcnm"
server_key: "ba384516d6e9d10bf5342b5fcc98d734"
server_key_iv: "43ccce4e0cd592d49b4a2ecd0c6635b1"
server_key_version: "encryptionkey@0"
I20260620 18:34:57.757131 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:57.762817 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:57.767360 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:57.767743 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:57.786931 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36955
I20260620 18:34:57.786970 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36955 every 8 connection(s)
I20260620 18:34:57.788367 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:57.792060 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:57.797901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 21987
I20260620 18:34:57.798080 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:57.798610 388 external_mini_cluster.cc:1428] Setting key 90126f3cfcc3fb21df1e0175e6b2fd1e
I20260620 18:34:57.802786 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d: Bootstrap starting.
I20260620 18:34:57.806316 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:57.800488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:42597 (local address 127.0.0.1:36955)
0620 18:34:57.801328 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:57.801339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:57.801379 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:34:57.803076 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:57.803097 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:57.803440 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:57.803817 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:57.804844 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:57.804850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:57.804862 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:57.804875 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:57.804933 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:57.804951 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:57.805118 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:57.805244 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:34:57.805443 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":384,"threads_started":1}
I20260620 18:34:57.806420 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:57.807772 22054 log.cc:824] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:57.810313 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:57.810693 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d: No bootstrap required, opened a new log
I20260620 18:34:57.817274 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:57.814791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:47939 (local address 127.0.0.1:36955)
0620 18:34:57.815003 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:57.815009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:57.815310 (+ 301us) server_negotiation.cc:407] Connection header received
0620 18:34:57.815448 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:57.815452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:57.815501 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:57.815667 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:57.816609 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:57.816613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:57.816616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:57.816630 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:57.816669 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:57.816684 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:57.816801 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:57.816911 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:34:57.816957 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:34:57.819284 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260620 18:34:57.819694 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:57.819816 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943c4595c7074bfcab02395f9e9a3d8d, State: Initialized, Role: FOLLOWER
I20260620 18:34:57.820434 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260620 18:34:57.820715 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:57.820870 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:57.821009 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:57.823143 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260620 18:34:57.823657 22054 leader_election.cc:302] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 943c4595c7074bfcab02395f9e9a3d8d; no voters:
I20260620 18:34:57.824275 22054 leader_election.cc:288] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:57.824388 22064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:57.825428 22064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [term 1 LEADER]: Becoming Leader. State: Replica: 943c4595c7074bfcab02395f9e9a3d8d, State: Running, Role: LEADER
I20260620 18:34:57.825973 22064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260620 18:34:57.826524 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:57.828966 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943c4595c7074bfcab02395f9e9a3d8d. Latest consensus state: current_term: 1 leader_uuid: "943c4595c7074bfcab02395f9e9a3d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260620 18:34:57.829434 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:57.829463 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "943c4595c7074bfcab02395f9e9a3d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943c4595c7074bfcab02395f9e9a3d8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260620 18:34:57.829897 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:57.833557 22072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:57.839931 22072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:57.848942 22072 catalog_manager.cc:1347] Generated new cluster ID: 03e7eec148cc42f5a1b07786d485dd43
I20260620 18:34:57.849154 22072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:57.888165 22072 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:57.888690 22072 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:57.901036 22072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 943c4595c7074bfcab02395f9e9a3d8d: Generated new TSK 0
I20260620 18:34:57.902006 22072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:57.911392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21987
W20260620 18:34:57.912667 22049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:42597
W20260620 18:34:57.913304 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 21987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:34:57.913450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 21987
2026-06-20T18:34:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:58 dist-test-slave-mcnm krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:58 dist-test-slave-mcnm krb5kdc[22085](info): set up 2 sockets
Jun 20 18:34:58 dist-test-slave-mcnm krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260620 18:34:59.031957 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:34:59 dist-test-slave-mcnm krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:34:59.277164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46487
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:59.603307 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:59.603685 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:59.603843 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:59.611130 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:59.611232 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:59.611363 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:59.611465 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:59.611549 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:59.621320 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:34:59.622618 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:59.624173 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:59.637317 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:59.637272 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:59.637857 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:59.638510 22098 server_base.cc:1034] running on GCE node
I20260620 18:34:59.639470 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:59.640951 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:59.642201 22098 hybrid_clock.cc:648] HybridClock initialized: now 1781980499642158 us; error 57 us; skew 500 ppm
I20260620 18:34:59.642575 22098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:59 dist-test-slave-mcnm krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:59.645902 22098 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:34:59.647471 22098 webserver.cc:460] Webserver started at http://127.0.97.62:44161/ using document root <none> and password file <none>
I20260620 18:34:59.648234 22098 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:59.648392 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:59.648725 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:59.651870 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "d86e4e4d311540fc811b4111b1de5984"
format_stamp: "Formatted at 2026-06-20 18:34:59 on dist-test-slave-mcnm"
server_key: "df3b6874e4e29d29d872fa615eca880a"
server_key_iv: "73ec529b88d388179d1d086033c4ed7b"
server_key_version: "encryptionkey@0"
I20260620 18:34:59.652688 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "d86e4e4d311540fc811b4111b1de5984"
format_stamp: "Formatted at 2026-06-20 18:34:59 on dist-test-slave-mcnm"
server_key: "df3b6874e4e29d29d872fa615eca880a"
server_key_iv: "73ec529b88d388179d1d086033c4ed7b"
server_key_version: "encryptionkey@0"
I20260620 18:34:59.657996 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:34:59.661440 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:59.662706 22098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:34:59.663017 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "d86e4e4d311540fc811b4111b1de5984"
format_stamp: "Formatted at 2026-06-20 18:34:59 on dist-test-slave-mcnm"
server_key: "df3b6874e4e29d29d872fa615eca880a"
server_key_iv: "73ec529b88d388179d1d086033c4ed7b"
server_key_version: "encryptionkey@0"
I20260620 18:34:59.663229 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:59.679104 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:59.683403 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:59.683724 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:59.702530 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46487
I20260620 18:34:59.702569 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46487 every 8 connection(s)
I20260620 18:34:59.703889 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:59.707505 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:59.710875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22098
I20260620 18:34:59.711073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:34:59.711642 388 external_mini_cluster.cc:1428] Setting key f511425ecec8b703f258d04b74e0a220
I20260620 18:34:59.720266 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984: Bootstrap starting.
I20260620 18:34:59.724120 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:59.725534 22165 log.cc:824] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:34:59 dist-test-slave-mcnm krb5kdc[22085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:34:59.728992 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984: No bootstrap required, opened a new log
I20260620 18:34:59.736004 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:59.713539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.97.62:46487)
0620 18:34:59.714317 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:59.714352 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:59.714395 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:34:59.716139 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:59.716162 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:59.716577 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:59.716915 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:59.718638 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.720031 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.721818 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.722562 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.722723 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.728195 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:59.728203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:59.728219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:59.728258 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:59.731568 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:59.732516 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:59.732523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:59.732529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:59.732696 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:59.733171 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:59.733175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:59.733177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:59.733724 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:59.733972 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:59.734485 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 18:34:59.734829 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":374,"threads_started":1}
I20260620 18:34:59.738277 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } }
I20260620 18:34:59.738722 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:59.738864 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86e4e4d311540fc811b4111b1de5984, State: Initialized, Role: FOLLOWER
I20260620 18:34:59.739552 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } }
I20260620 18:34:59.739809 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:59.739979 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:59.740154 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:59.740325 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:59.742300 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } }
I20260620 18:34:59.742776 22165 leader_election.cc:302] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86e4e4d311540fc811b4111b1de5984; no voters:
I20260620 18:34:59.743462 22165 leader_election.cc:288] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:59.743634 22172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:59.744894 22172 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [term 1 LEADER]: Becoming Leader. State: Replica: d86e4e4d311540fc811b4111b1de5984, State: Running, Role: LEADER
I20260620 18:34:59.745755 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:59.745849 22172 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } }
I20260620 18:34:59.748726 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86e4e4d311540fc811b4111b1de5984. Latest consensus state: current_term: 1 leader_uuid: "d86e4e4d311540fc811b4111b1de5984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } } }
I20260620 18:34:59.749241 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:59.749760 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86e4e4d311540fc811b4111b1de5984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86e4e4d311540fc811b4111b1de5984" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46487 } } }
I20260620 18:34:59.750073 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:59.755304 22182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:59.762107 22182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:59.765748 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:59.745410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52527 (local address 127.0.97.62:46487)
0620 18:34:59.745614 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:59.745633 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:59.745779 (+ 146us) server_negotiation.cc:407] Connection header received
0620 18:34:59.746034 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:59.746040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:59.746119 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:59.746219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:59.748353 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.749818 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.752014 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.752924 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.753042 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.756578 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:59.756583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:59.756587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:59.756617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:59.760934 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:59.761711 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:59.761717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:59.761721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:59.761883 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:59.762366 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:59.762371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:59.762374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:59.762928 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:59.764699 (+ 1771us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:59.764948 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:34:59.765200 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:34:59.771479 22182 catalog_manager.cc:1347] Generated new cluster ID: 9df9976a133b45679b33c471b25e304e
I20260620 18:34:59.771636 22182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:59.793166 22182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:59.794486 22182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:59.802160 22182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d86e4e4d311540fc811b4111b1de5984: Generated new TSK 0
I20260620 18:34:59.803066 22182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:59.810040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22098
I20260620 18:34:59.811666 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:59.901829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22098
W20260620 18:34:59.918941 22083 connection.cc:535] client connection to 127.0.97.62:46487 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:46487): Connection reset by peer (error 104)
2026-06-20T18:34:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:00 dist-test-slave-mcnm krb5kdc[22198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:00 dist-test-slave-mcnm krb5kdc[22198](info): set up 2 sockets
Jun 20 18:35:00 dist-test-slave-mcnm krb5kdc[22198](info): commencing operation
krb5kdc: starting...
W20260620 18:35:00.999883 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:01 dist-test-slave-mcnm krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:01.250936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:01.576165 22211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:01.576444 22211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:01.576536 22211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:01.583719 22211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:01.583823 22211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:01.583870 22211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:01.583935 22211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:01.583981 22211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:01.593663 22211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:01.594862 22211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:01.596122 22211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:01.608006 22217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:01.608088 22219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:01.608377 22216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:01.609587 22211 server_base.cc:1034] running on GCE node
I20260620 18:35:01.610487 22211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:01.612159 22211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:01.613396 22211 hybrid_clock.cc:648] HybridClock initialized: now 1781980501613346 us; error 55 us; skew 500 ppm
I20260620 18:35:01.613767 22211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:01 dist-test-slave-mcnm krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:01.617292 22211 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:01.618919 22211 webserver.cc:460] Webserver started at http://127.0.97.62:37943/ using document root <none> and password file <none>
I20260620 18:35:01.619728 22211 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:01.619879 22211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:01.620200 22211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:01.623322 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "e8afe83c4a0040f98b0036c332907ce0"
format_stamp: "Formatted at 2026-06-20 18:35:01 on dist-test-slave-mcnm"
server_key: "3edff4f74e417d8a6330f488941d01f9"
server_key_iv: "918c36e580b507ca1ed2caf14fecbe56"
server_key_version: "encryptionkey@0"
I20260620 18:35:01.624221 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8afe83c4a0040f98b0036c332907ce0"
format_stamp: "Formatted at 2026-06-20 18:35:01 on dist-test-slave-mcnm"
server_key: "3edff4f74e417d8a6330f488941d01f9"
server_key_iv: "918c36e580b507ca1ed2caf14fecbe56"
server_key_version: "encryptionkey@0"
I20260620 18:35:01.629608 22211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:35:01.633176 22226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:01.634573 22211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:35:01.634770 22211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "e8afe83c4a0040f98b0036c332907ce0"
format_stamp: "Formatted at 2026-06-20 18:35:01 on dist-test-slave-mcnm"
server_key: "3edff4f74e417d8a6330f488941d01f9"
server_key_iv: "918c36e580b507ca1ed2caf14fecbe56"
server_key_version: "encryptionkey@0"
I20260620 18:35:01.634963 22211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:01.653769 22211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:01.658232 22211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:01.658561 22211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:01.678535 22211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44411
I20260620 18:35:01.678620 22277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44411 every 8 connection(s)
I20260620 18:35:01.679981 22211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:01.683447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22211
I20260620 18:35:01.683653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:01.684158 22278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:01.684209 388 external_mini_cluster.cc:1428] Setting key 14f5dedd646b57a0491adea2be372bd3
I20260620 18:35:01.698002 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0: Bootstrap starting.
I20260620 18:35:01.698233 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:01.686026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50413 (local address 127.0.97.62:44411)
0620 18:35:01.686878 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:01.686891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:01.686936 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:35:01.688987 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:01.689012 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:01.689426 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:01.689788 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:01.690353 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.691894 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.693680 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.694369 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.694503 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.695856 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:01.695864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:01.695878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:01.696447 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:01.696530 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:01.696555 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:01.696677 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:01.697064 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:35:01.697252 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":391,"threads_started":1}
I20260620 18:35:01.701239 22278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:01.701902 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:01.702404 22278 log.cc:824] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:01.705754 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0: No bootstrap required, opened a new log
I20260620 18:35:01.714087 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:01.707052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38029 (local address 127.0.97.62:44411)
0620 18:35:01.707235 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:01.707243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:01.707784 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:35:01.707970 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:01.707976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:01.708060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:01.708158 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:01.708656 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.709852 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.711266 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.712000 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.712130 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.713238 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:01.713250 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:01.713254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:01.713277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:01.713332 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:01.713353 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:01.713472 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:01.713640 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:35:01.713703 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:35:01.715477 22278 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } }
I20260620 18:35:01.715883 22278 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:01.715999 22278 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8afe83c4a0040f98b0036c332907ce0, State: Initialized, Role: FOLLOWER
I20260620 18:35:01.716598 22278 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } }
I20260620 18:35:01.716835 22278 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:01.716949 22278 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:01.717079 22278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:01.719710 22278 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } }
I20260620 18:35:01.720175 22278 leader_election.cc:302] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8afe83c4a0040f98b0036c332907ce0; no voters:
I20260620 18:35:01.720685 22278 leader_election.cc:288] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:01.720832 22288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:01.721558 22288 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [term 1 LEADER]: Becoming Leader. State: Replica: e8afe83c4a0040f98b0036c332907ce0, State: Running, Role: LEADER
I20260620 18:35:01.722078 22288 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } }
I20260620 18:35:01.722863 22278 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:01.725173 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8afe83c4a0040f98b0036c332907ce0. Latest consensus state: current_term: 1 leader_uuid: "e8afe83c4a0040f98b0036c332907ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } } }
I20260620 18:35:01.725616 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:01.726827 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:01.726809 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8afe83c4a0040f98b0036c332907ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8afe83c4a0040f98b0036c332907ce0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44411 } } }
I20260620 18:35:01.727223 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:01.732115 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:01.740445 22295 catalog_manager.cc:1347] Generated new cluster ID: 499d9f62bfc14c3e98087178d3801ae1
I20260620 18:35:01.740574 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:01.751964 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:01.753707 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:01.772078 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8afe83c4a0040f98b0036c332907ce0: Generated new TSK 0
I20260620 18:35:01.773144 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:01.785799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22211
W20260620 18:35:01.787130 22273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50413
W20260620 18:35:01.787966 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22211: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:01.788120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22211
W20260620 18:35:01.805830 22196 connection.cc:535] client connection to 127.0.97.62:44411 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44411): Connection reset by peer (error 104)
2026-06-20T18:35:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:01 dist-test-slave-mcnm krb5kdc[22309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:01 dist-test-slave-mcnm krb5kdc[22309](info): set up 2 sockets
Jun 20 18:35:01 dist-test-slave-mcnm krb5kdc[22309](info): commencing operation
krb5kdc: starting...
W20260620 18:35:02.928145 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:03 dist-test-slave-mcnm krb5kdc[22309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:03.184193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33987
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:03.527904 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:03.528194 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:03.528364 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:03.535897 22322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:03.536010 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:03.536108 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:03.536180 22322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:03.536250 22322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:03.546212 22322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:03.547634 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:03.548941 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:03.560751 22327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:03.560709 22330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:03.560810 22328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:03.561525 22322 server_base.cc:1034] running on GCE node
I20260620 18:35:03.562328 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:03.563743 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:03.564965 22322 hybrid_clock.cc:648] HybridClock initialized: now 1781980503564927 us; error 51 us; skew 500 ppm
I20260620 18:35:03.565317 22322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:03 dist-test-slave-mcnm krb5kdc[22309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:03.568351 22322 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:03.569808 22322 webserver.cc:460] Webserver started at http://127.0.97.62:42331/ using document root <none> and password file <none>
I20260620 18:35:03.570473 22322 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:03.570613 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:03.570919 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:03.573872 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "6fd7287fd979402aa13fae23b8ed86bf"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-mcnm"
server_key: "56beb7c8a82104e132b578f50e4180f4"
server_key_iv: "ff267a6819ec2cf8144ab65342e54953"
server_key_version: "encryptionkey@0"
I20260620 18:35:03.574638 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "6fd7287fd979402aa13fae23b8ed86bf"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-mcnm"
server_key: "56beb7c8a82104e132b578f50e4180f4"
server_key_iv: "ff267a6819ec2cf8144ab65342e54953"
server_key_version: "encryptionkey@0"
I20260620 18:35:03.579707 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:35:03.583187 22337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:03.584643 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:03.584832 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "6fd7287fd979402aa13fae23b8ed86bf"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-mcnm"
server_key: "56beb7c8a82104e132b578f50e4180f4"
server_key_iv: "ff267a6819ec2cf8144ab65342e54953"
server_key_version: "encryptionkey@0"
I20260620 18:35:03.585036 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:03.590687 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:03.595598 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:03.595921 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:03.614311 22322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33987
I20260620 18:35:03.614347 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33987 every 8 connection(s)
I20260620 18:35:03.615954 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:03.618408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22322
I20260620 18:35:03.618593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:03.619060 388 external_mini_cluster.cc:1428] Setting key 7c949de2820b2ecb189f52df246baade
I20260620 18:35:03.620240 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:03.627714 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.620847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49013 (local address 127.0.97.62:33987)
0620 18:35:03.621630 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.621645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.621698 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:35:03.623678 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.623705 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.624159 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.624616 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.625625 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:03.625634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:03.625650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:03.625685 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:03.625769 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:03.625794 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:03.626031 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:03.626169 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:35:03.626370 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":315,"threads_started":1}
I20260620 18:35:03.631718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:03.635004 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf: Bootstrap starting.
I20260620 18:35:03.638909 22389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:03.640014 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.637270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37357 (local address 127.0.97.62:33987)
0620 18:35:03.637466 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.637473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.637952 (+ 479us) server_negotiation.cc:407] Connection header received
0620 18:35:03.638109 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.638115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.638179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.638291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.639274 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:03.639284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:03.639291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:03.639312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:03.639369 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:03.639397 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:03.639544 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:03.639663 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 18:35:03.639716 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:03.640341 22389 log.cc:824] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:03.643155 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf: No bootstrap required, opened a new log
I20260620 18:35:03.650314 22389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } }
I20260620 18:35:03.650673 22389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:03.650748 22389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd7287fd979402aa13fae23b8ed86bf, State: Initialized, Role: FOLLOWER
I20260620 18:35:03.651275 22389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } }
I20260620 18:35:03.651465 22389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:03.651618 22389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:03.651749 22389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:03.654141 22389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } }
I20260620 18:35:03.654568 22389 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd7287fd979402aa13fae23b8ed86bf; no voters:
I20260620 18:35:03.655133 22389 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:03.655340 22399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:03.656107 22399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [term 1 LEADER]: Becoming Leader. State: Replica: 6fd7287fd979402aa13fae23b8ed86bf, State: Running, Role: LEADER
I20260620 18:35:03.656737 22399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } }
I20260620 18:35:03.657079 22389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:03.659884 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd7287fd979402aa13fae23b8ed86bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } } }
I20260620 18:35:03.660295 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:03.660041 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd7287fd979402aa13fae23b8ed86bf. Latest consensus state: current_term: 1 leader_uuid: "6fd7287fd979402aa13fae23b8ed86bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7287fd979402aa13fae23b8ed86bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33987 } } }
I20260620 18:35:03.660426 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:03.662252 22406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:03.667117 22406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:03.675592 22406 catalog_manager.cc:1347] Generated new cluster ID: c5bb24a7b7ba4cf2ada3b1de6eb9d840
I20260620 18:35:03.675716 22406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:03.726365 22406 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:03.726867 22406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:03.743156 22406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd7287fd979402aa13fae23b8ed86bf: Generated new TSK 0
I20260620 18:35:03.744275 22406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:03.755849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22322
W20260620 18:35:03.756898 22384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49013
W20260620 18:35:03.757614 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:03.757757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22322
2026-06-20T18:35:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:03 dist-test-slave-mcnm krb5kdc[22420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:03 dist-test-slave-mcnm krb5kdc[22420](info): set up 2 sockets
Jun 20 18:35:03 dist-test-slave-mcnm krb5kdc[22420](info): commencing operation
krb5kdc: starting...
W20260620 18:35:04.876116 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:05.134976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:05.464048 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:05.464336 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:05.464497 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:05.471808 22433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:05.471935 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:05.472018 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:05.472158 22433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:05.472211 22433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:05.482197 22433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:05.483474 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:05.484771 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:05.496438 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:05.496507 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:05.496527 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:05.497303 22433 server_base.cc:1034] running on GCE node
I20260620 18:35:05.498178 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:05.499646 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:05.500895 22433 hybrid_clock.cc:648] HybridClock initialized: now 1781980505500833 us; error 58 us; skew 500 ppm
I20260620 18:35:05.501343 22433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:05.504637 22433 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:05.506094 22433 webserver.cc:460] Webserver started at http://127.0.97.62:34987/ using document root <none> and password file <none>
I20260620 18:35:05.506780 22433 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:05.506923 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:05.507228 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:05.510242 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "653878273a4847c79a29869252801d87"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-mcnm"
server_key: "a7dfa2bb7184f1ade6906e06be234017"
server_key_iv: "baf08c267366bc39016fcb71b86983ae"
server_key_version: "encryptionkey@0"
I20260620 18:35:05.510996 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "653878273a4847c79a29869252801d87"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-mcnm"
server_key: "a7dfa2bb7184f1ade6906e06be234017"
server_key_iv: "baf08c267366bc39016fcb71b86983ae"
server_key_version: "encryptionkey@0"
I20260620 18:35:05.516256 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:35:05.519917 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:05.521359 22433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:35:05.521564 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "653878273a4847c79a29869252801d87"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-mcnm"
server_key: "a7dfa2bb7184f1ade6906e06be234017"
server_key_iv: "baf08c267366bc39016fcb71b86983ae"
server_key_version: "encryptionkey@0"
I20260620 18:35:05.521785 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:05.541213 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:05.545542 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:05.545845 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:05.565881 22433 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40731
I20260620 18:35:05.565922 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40731 every 8 connection(s)
I20260620 18:35:05.567333 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:05.569142 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22433
I20260620 18:35:05.569352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:05.569789 388 external_mini_cluster.cc:1428] Setting key 8df588915baedb87ccba442c94096a3d
I20260620 18:35:05.571813 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:35:05.585466 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87: Bootstrap starting.
I20260620 18:35:05.589887 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:05.591146 22500 log.cc:824] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:05.594776 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87: No bootstrap required, opened a new log
I20260620 18:35:05.595005 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.571478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43737 (local address 127.0.97.62:40731)
0620 18:35:05.572299 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.572310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.572350 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:35:05.574004 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.574025 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.574412 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.574737 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.576767 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.578299 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.580110 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.580821 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.580971 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.585986 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:05.585997 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:05.586017 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:05.586071 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:05.590273 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.591075 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.591085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.591097 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.591350 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.591848 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.591856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.591862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.592447 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:05.592713 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:05.593206 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 18:35:05.593648 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":368,"threads_started":1}
I20260620 18:35:05.598722 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:05.602936 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } }
I20260620 18:35:05.603392 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:05.603500 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653878273a4847c79a29869252801d87, State: Initialized, Role: FOLLOWER
I20260620 18:35:05.604214 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } }
I20260620 18:35:05.604447 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:05.604591 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:05.604766 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:05.607482 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } }
I20260620 18:35:05.608081 22500 leader_election.cc:302] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 653878273a4847c79a29869252801d87; no voters:
I20260620 18:35:05.608698 22500 leader_election.cc:288] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:05.608812 22510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:05.609953 22510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [term 1 LEADER]: Becoming Leader. State: Replica: 653878273a4847c79a29869252801d87, State: Running, Role: LEADER
I20260620 18:35:05.610636 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } }
I20260620 18:35:05.610831 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:05.613533 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653878273a4847c79a29869252801d87. Latest consensus state: current_term: 1 leader_uuid: "653878273a4847c79a29869252801d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } } }
I20260620 18:35:05.613921 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:05.614455 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "653878273a4847c79a29869252801d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653878273a4847c79a29869252801d87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40731 } } }
I20260620 18:35:05.614769 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:05.617018 22516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:05.620723 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.603008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57603 (local address 127.0.97.62:40731)
0620 18:35:05.603195 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.603200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.603630 (+ 430us) server_negotiation.cc:407] Connection header received
0620 18:35:05.603806 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.603812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.603894 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.603988 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.605506 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.606637 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.608029 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.608783 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.608910 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.612309 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:05.612318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:05.612326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:05.612382 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:05.616533 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.617540 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.617549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.617555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.617826 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.618377 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.618385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.618391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.618875 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:05.619216 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:05.619922 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:35:05.620210 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:35:05.627254 22516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:05.636175 22516 catalog_manager.cc:1347] Generated new cluster ID: dbbcf33e75344210a734e0f74d5b4ecd
I20260620 18:35:05.636299 22516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:05.654314 22516 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:05.655591 22516 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:05.662326 22516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 653878273a4847c79a29869252801d87: Generated new TSK 0
I20260620 18:35:05.663317 22516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:05.680195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22433
I20260620 18:35:05.681545 22495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:05.772629 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22433
2026-06-20T18:35:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22533](info): set up 2 sockets
Jun 20 18:35:05 dist-test-slave-mcnm krb5kdc[22533](info): commencing operation
krb5kdc: starting...
W20260620 18:35:06.896070 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:07 dist-test-slave-mcnm krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:07.146245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44827
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:07.470621 22546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:07.470957 22546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:07.471119 22546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:07.478294 22546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:07.478435 22546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:07.478534 22546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:07.478636 22546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:07.478716 22546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:07.488406 22546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:07.489636 22546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:07.490942 22546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:07.502823 22551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.502864 22552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.502912 22554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:07.504196 22546 server_base.cc:1034] running on GCE node
I20260620 18:35:07.504987 22546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:07.506448 22546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:07.507664 22546 hybrid_clock.cc:648] HybridClock initialized: now 1781980507507627 us; error 44 us; skew 500 ppm
I20260620 18:35:07.508020 22546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:07 dist-test-slave-mcnm krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:07.511201 22546 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:07.512794 22546 webserver.cc:460] Webserver started at http://127.0.97.62:40361/ using document root <none> and password file <none>
I20260620 18:35:07.513482 22546 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:07.513624 22546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:07.513947 22546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:07.516992 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "854583cb2d9d4b57a991076563eee771"
format_stamp: "Formatted at 2026-06-20 18:35:07 on dist-test-slave-mcnm"
server_key: "785c4ad44533a3c70524056e757aad83"
server_key_iv: "fd0772d64c880682c6d962575fe754da"
server_key_version: "encryptionkey@0"
I20260620 18:35:07.517720 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "854583cb2d9d4b57a991076563eee771"
format_stamp: "Formatted at 2026-06-20 18:35:07 on dist-test-slave-mcnm"
server_key: "785c4ad44533a3c70524056e757aad83"
server_key_iv: "fd0772d64c880682c6d962575fe754da"
server_key_version: "encryptionkey@0"
I20260620 18:35:07.522761 22546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:35:07.526243 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:07.527611 22546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:35:07.527773 22546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "854583cb2d9d4b57a991076563eee771"
format_stamp: "Formatted at 2026-06-20 18:35:07 on dist-test-slave-mcnm"
server_key: "785c4ad44533a3c70524056e757aad83"
server_key_iv: "fd0772d64c880682c6d962575fe754da"
server_key_version: "encryptionkey@0"
I20260620 18:35:07.527974 22546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:07.540328 22546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:07.544677 22546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:07.544960 22546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:07.563263 22546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44827
I20260620 18:35:07.563293 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44827 every 8 connection(s)
I20260620 18:35:07.564720 22546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:07.568332 22613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:07.569465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22546
I20260620 18:35:07.569671 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:07.570158 388 external_mini_cluster.cc:1428] Setting key 527660fe6f1989ed2f0e2f445f5087a9
I20260620 18:35:07.581853 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771: Bootstrap starting.
I20260620 18:35:07.583773 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:07.571973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57927 (local address 127.0.97.62:44827)
0620 18:35:07.572803 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:07.572815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:07.572865 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:35:07.574548 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:07.574570 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:07.575022 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:07.575367 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:07.575938 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.577249 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.579097 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.579949 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.580098 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.581409 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:07.581417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:07.581430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:07.582019 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:07.582114 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:07.582149 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:07.582256 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:07.582463 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:35:07.582762 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":411,"threads_started":1}
I20260620 18:35:07.585435 22613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:07.586583 22613 log.cc:824] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:07.587791 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:07.589779 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771: No bootstrap required, opened a new log
I20260620 18:35:07.599269 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:07.592876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45311 (local address 127.0.97.62:44827)
0620 18:35:07.593060 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:07.593072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:07.593487 (+ 415us) server_negotiation.cc:407] Connection header received
0620 18:35:07.593651 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:07.593656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:07.593739 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:07.593841 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:07.594289 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.595430 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.596790 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.597473 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.597599 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.598580 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:07.598586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:07.598591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:07.598617 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:07.598664 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:07.598683 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:07.598764 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:07.598896 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 18:35:07.598981 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:35:07.599702 22613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 18:35:07.600108 22613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:07.600215 22613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854583cb2d9d4b57a991076563eee771, State: Initialized, Role: FOLLOWER
I20260620 18:35:07.600816 22613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 18:35:07.601058 22613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:07.601202 22613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:07.601362 22613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:07.603431 22613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 18:35:07.603976 22613 leader_election.cc:302] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 854583cb2d9d4b57a991076563eee771; no voters:
I20260620 18:35:07.604530 22613 leader_election.cc:288] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:07.604671 22623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:07.605383 22623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [term 1 LEADER]: Becoming Leader. State: Replica: 854583cb2d9d4b57a991076563eee771, State: Running, Role: LEADER
I20260620 18:35:07.605998 22623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 18:35:07.606725 22613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:07.608927 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 854583cb2d9d4b57a991076563eee771. Latest consensus state: current_term: 1 leader_uuid: "854583cb2d9d4b57a991076563eee771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } } }
I20260620 18:35:07.608827 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "854583cb2d9d4b57a991076563eee771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854583cb2d9d4b57a991076563eee771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } } }
I20260620 18:35:07.609268 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:07.609273 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:07.612171 22630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:07.617384 22630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:07.627140 22630 catalog_manager.cc:1347] Generated new cluster ID: 66f10a9e85404e73ad071fc41296f4c4
I20260620 18:35:07.627270 22630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:07.642774 22630 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:07.644758 22630 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:07.655967 22630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 854583cb2d9d4b57a991076563eee771: Generated new TSK 0
I20260620 18:35:07.657030 22630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:07.668833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22546
W20260620 18:35:07.669947 22608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57927
W20260620 18:35:07.670617 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:07.670755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22546
2026-06-20T18:35:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:07 dist-test-slave-mcnm krb5kdc[22644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:07 dist-test-slave-mcnm krb5kdc[22644](info): set up 2 sockets
Jun 20 18:35:07 dist-test-slave-mcnm krb5kdc[22644](info): commencing operation
krb5kdc: starting...
W20260620 18:35:08.772327 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:08 dist-test-slave-mcnm krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:09.036490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35497
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:09.365583 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.365909 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.366082 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.373224 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:09.373369 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.373482 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.373590 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:09.373667 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:09.384626 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:09.385905 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.387223 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.398810 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.398844 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.398975 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.399616 22657 server_base.cc:1034] running on GCE node
I20260620 18:35:09.400388 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.401810 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.403080 22657 hybrid_clock.cc:648] HybridClock initialized: now 1781980509403025 us; error 53 us; skew 500 ppm
I20260620 18:35:09.403463 22657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:09 dist-test-slave-mcnm krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:09.411952 22657 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:09.413645 22657 webserver.cc:460] Webserver started at http://127.0.97.62:40945/ using document root <none> and password file <none>
I20260620 18:35:09.414371 22657 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.414517 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.414831 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.418043 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "a4763a75bd2742d9acc091a407238867"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-mcnm"
server_key: "d020f603791907570c3ae26530eb70b4"
server_key_iv: "b58609aebdced694a1880a70ebb25d69"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.418851 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "a4763a75bd2742d9acc091a407238867"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-mcnm"
server_key: "d020f603791907570c3ae26530eb70b4"
server_key_iv: "b58609aebdced694a1880a70ebb25d69"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.424589 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 18:35:09.428279 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.429656 22657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:09.429849 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "a4763a75bd2742d9acc091a407238867"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-mcnm"
server_key: "d020f603791907570c3ae26530eb70b4"
server_key_iv: "b58609aebdced694a1880a70ebb25d69"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.430105 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.435894 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.439982 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.440282 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.459018 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35497
I20260620 18:35:09.459066 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35497 every 8 connection(s)
I20260620 18:35:09.460562 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:09.460911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22657
I20260620 18:35:09.461122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:09.461719 388 external_mini_cluster.cc:1428] Setting key fa0adc2953332d7d2610c84f1ac15a9e
I20260620 18:35:09.465044 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:09.469521 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.463564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41297 (local address 127.0.97.62:35497)
0620 18:35:09.464369 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.464381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.464422 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:35:09.466144 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.466166 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.466482 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.466794 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.467846 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.467854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.467867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.467881 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.467957 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.467977 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.468154 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.468261 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.468619 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":360,"threads_started":1}
I20260620 18:35:09.472995 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:09.477571 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867: Bootstrap starting.
I20260620 18:35:09.480674 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.478075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50239 (local address 127.0.97.62:35497)
0620 18:35:09.478285 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.478294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.478836 (+ 542us) server_negotiation.cc:407] Connection header received
0620 18:35:09.478956 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.478962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.479033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.479175 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.480036 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.480042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.480046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.480066 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.480115 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.480134 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.480251 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.480320 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.480380 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:35:09.481326 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:09.482429 22724 log.cc:824] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:09.485062 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867: No bootstrap required, opened a new log
I20260620 18:35:09.492506 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } }
I20260620 18:35:09.492878 22724 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:09.492987 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4763a75bd2742d9acc091a407238867, State: Initialized, Role: FOLLOWER
I20260620 18:35:09.493530 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } }
I20260620 18:35:09.493762 22724 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:09.493899 22724 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:09.494076 22724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:09.496943 22724 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } }
I20260620 18:35:09.497445 22724 leader_election.cc:302] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4763a75bd2742d9acc091a407238867; no voters:
I20260620 18:35:09.498003 22724 leader_election.cc:288] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:09.498128 22734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:09.498826 22734 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [term 1 LEADER]: Becoming Leader. State: Replica: a4763a75bd2742d9acc091a407238867, State: Running, Role: LEADER
I20260620 18:35:09.499364 22734 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } }
I20260620 18:35:09.500137 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:09.502615 22736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4763a75bd2742d9acc091a407238867. Latest consensus state: current_term: 1 leader_uuid: "a4763a75bd2742d9acc091a407238867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } } }
I20260620 18:35:09.502652 22735 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4763a75bd2742d9acc091a407238867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4763a75bd2742d9acc091a407238867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35497 } } }
I20260620 18:35:09.502975 22735 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:09.502974 22736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:09.506376 22742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:09.512339 22742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:09.521185 22742 catalog_manager.cc:1347] Generated new cluster ID: 7bbde51be7274bcc9bc72c82ccab4e76
I20260620 18:35:09.521355 22742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:09.534894 22742 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:09.535341 22742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:09.544178 22742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4763a75bd2742d9acc091a407238867: Generated new TSK 0
I20260620 18:35:09.545018 22742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:09.558743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22657
W20260620 18:35:09.560400 22719 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41297
W20260620 18:35:09.561108 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22657: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:09.561264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22657
2026-06-20T18:35:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1890 ms)
[----------] 18 tests from AuthTokenIssuingTest (35092 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:09 dist-test-slave-mcnm krb5kdc[22755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:09 dist-test-slave-mcnm krb5kdc[22755](info): set up 2 sockets
Jun 20 18:35:09 dist-test-slave-mcnm krb5kdc[22755](info): commencing operation
krb5kdc: starting...
W20260620 18:35:10.696233 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:10 dist-test-slave-mcnm krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:10.968043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:11.299216 22768 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:11.299650 22768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.299779 22768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.299913 22768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:11.307574 22768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:11.307713 22768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.307809 22768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.307906 22768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:11.307994 22768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:11.318140 22768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:11.319442 22768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.320806 22768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.332336 22773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.332397 22774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.332441 22776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.333570 22768 server_base.cc:1034] running on GCE node
I20260620 18:35:11.334370 22768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.335760 22768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.336997 22768 hybrid_clock.cc:648] HybridClock initialized: now 1781980511336940 us; error 36 us; skew 500 ppm
I20260620 18:35:11.337316 22768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:11 dist-test-slave-mcnm krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.340747 22768 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:11.342029 22768 webserver.cc:460] Webserver started at http://127.0.97.62:45645/ using document root <none> and password file <none>
I20260620 18:35:11.342650 22768 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.342733 22768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.342976 22768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.345851 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "450664b3449045319063e68c94b4bbf4"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "c9b1c10fd8022bc21dbd48e10ed7e3b0"
server_key_iv: "2bca33c2dcc2e36ed2fd75239bd627a9"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.346558 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "450664b3449045319063e68c94b4bbf4"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "c9b1c10fd8022bc21dbd48e10ed7e3b0"
server_key_iv: "2bca33c2dcc2e36ed2fd75239bd627a9"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.351358 22768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 18:35:11.354895 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.356317 22768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:11.356500 22768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "450664b3449045319063e68c94b4bbf4"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "c9b1c10fd8022bc21dbd48e10ed7e3b0"
server_key_iv: "2bca33c2dcc2e36ed2fd75239bd627a9"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.356709 22768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.376688 22768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.380988 22768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.381265 22768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.401082 22768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35801
I20260620 18:35:11.401122 22834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35801 every 8 connection(s)
I20260620 18:35:11.402448 22768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:11.405875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22768
I20260620 18:35:11.406121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:11.406917 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.407038 388 external_mini_cluster.cc:1428] Setting key e39beb25f22801e8379762cb24fdc99a
I20260620 18:35:11.418336 22835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:11 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 18:35:11.443280 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.421919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.0.97.62:35801)
0620 18:35:11.423662 (+ 1743us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.423672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.423723 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:35:11.425548 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.425559 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.425859 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.426067 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.427334 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.429337 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.430853 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.431875 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.432096 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.435483 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.435492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.435505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.435602 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.438852 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.439581 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.439599 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.439601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.439741 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.440137 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.440143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.440146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.440639 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.440842 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.441398 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.441791 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1476,"spinlock_wait_cycles":35072,"thread_start_us":1275,"threads_started":1}
I20260620 18:35:11.443300 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.421920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35801 (local address 127.0.0.1:44334)
0620 18:35:11.422980 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.423087 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.425158 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.426128 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.426134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.426739 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.427110 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.427128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.429588 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.429592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.430703 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.430715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.432119 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.432124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.432257 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.432992 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.433018 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.435225 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.438999 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.439011 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.439024 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.439419 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.439860 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.439864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.439868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.440016 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.440777 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.440787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.441016 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.441268 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.441902 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":195,"thread_start_us":423,"threads_started":1}
I20260620 18:35:11.446800 22835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:11.448050 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.447670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40236)
0620 18:35:11.447845 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.447943 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 18:35:11.448457 22786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
I20260620 18:35:11.449998 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.449692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40240)
0620 18:35:11.449825 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.449886 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:11.450294 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45481: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 18:35:11.471225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45481
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:11.499696 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } attempt: 1
I20260620 18:35:11.500851 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.500429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40244)
0620 18:35:11.500627 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.500712 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:35:11.501791 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.501487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40260)
0620 18:35:11.501616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.501676 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:35:11.501977 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45481: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
I20260620 18:35:11.570277 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } attempt: 2
I20260620 18:35:11.571672 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.571038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40274)
0620 18:35:11.571356 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.571446 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 18:35:11.572963 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.572525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40286)
0620 18:35:11.572745 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.572833 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260620 18:35:11.573289 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45481: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
I20260620 18:35:11.664563 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } attempt: 3
I20260620 18:35:11.665812 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.665333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40292)
0620 18:35:11.665558 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.665657 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:35:11.666999 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.666640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40296)
0620 18:35:11.666790 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.666866 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:35:11.667241 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45481: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
W20260620 18:35:11.802852 22842 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:11.803256 22842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.803385 22842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.803557 22842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:35:11.810504 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } attempt: 4
W20260620 18:35:11.810968 22842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:11.811091 22842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.811204 22842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.811290 22842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:11.811398 22842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:11.811883 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.811329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40308)
0620 18:35:11.811680 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.811763 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260620 18:35:11.812829 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.812465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40312)
0620 18:35:11.812618 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.812701 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:35:11.812995 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45481: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45481: connect: Connection refused (error 111)
I20260620 18:35:11.821686 22842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:11.822988 22842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.824452 22842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.836499 22847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.836452 22850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.836421 22848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.837633 22842 server_base.cc:1034] running on GCE node
I20260620 18:35:11.838488 22842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.840010 22842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.841285 22842 hybrid_clock.cc:648] HybridClock initialized: now 1781980511841219 us; error 59 us; skew 500 ppm
I20260620 18:35:11.841670 22842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:11 dist-test-slave-mcnm krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.845582 22842 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 18:35:11.847566 22842 webserver.cc:460] Webserver started at http://127.0.97.61:34473/ using document root <none> and password file <none>
I20260620 18:35:11.848556 22842 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.848682 22842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.849119 22842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.853071 22842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data/instance:
uuid: "d15843def5ec49b880b9eb425596af03"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "872d850d5fc6bf5c40febe376a329809"
server_key_iv: "d80c7181cf586c4008d6c651ee41b821"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.853813 22842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal/instance:
uuid: "d15843def5ec49b880b9eb425596af03"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "872d850d5fc6bf5c40febe376a329809"
server_key_iv: "d80c7181cf586c4008d6c651ee41b821"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.859437 22842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:35:11.863615 22857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.865118 22842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 18:35:11.865314 22842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "d15843def5ec49b880b9eb425596af03"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-mcnm"
server_key: "872d850d5fc6bf5c40febe376a329809"
server_key_iv: "d80c7181cf586c4008d6c651ee41b821"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.865567 22842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.879307 22842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.883728 22842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.884048 22842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.903800 22842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45481
I20260620 18:35:11.903820 22908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45481 every 8 connection(s)
I20260620 18:35:11.905313 22842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
I20260620 18:35:11.905983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22842
I20260620 18:35:11.906209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-1/wal/instance
I20260620 18:35:11.906778 388 external_mini_cluster.cc:1428] Setting key ad07af2775ec95766ad4941d4018b223
I20260620 18:35:11.909595 22909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.917030 22909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 18:35:11 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:35:11.941727 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.919967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.0.97.62:35801)
0620 18:35:11.920167 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.920173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.921642 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 18:35:11.923819 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.923826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.923927 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.924062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.925910 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.926656 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.928458 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.928893 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.929035 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.934066 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.934075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.934084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.934141 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.937875 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.938750 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.938758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.938764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.938921 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.939582 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.939590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.939596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.940008 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.940173 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.940994 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.941273 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:35:11.942808 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.920578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35801 (local address 127.0.0.1:44348)
0620 18:35:11.921405 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.921452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.923440 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.924313 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.924323 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.924924 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.925692 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.925717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.926833 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.926837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.928307 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.928320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.929144 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.929152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.929380 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.930159 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.930183 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.933792 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.938069 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.938078 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.938093 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.938599 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.939194 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.939198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.939201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.939410 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.940161 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.940167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.940571 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.940827 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.941256 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":362,"threads_started":1}
I20260620 18:35:11.944633 22909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:11 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:11.963104 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.945411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40320)
0620 18:35:11.945761 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.945787 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.945911 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.946864 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.946870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.947409 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.947668 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.947682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.949605 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.949613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.951131 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.951148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.952107 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.952116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.952362 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.952998 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.953026 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.955744 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.960446 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.960450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.960453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.960865 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.961320 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.961327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.961331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.961456 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.962064 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.962068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.962237 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.962397 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.962698 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:35:11.963279 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.945735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.97.61:45481)
0620 18:35:11.946400 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.946407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.946430 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:11.946492 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.946498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.946685 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.946805 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.947852 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.949450 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.951271 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.951933 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.952031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.955986 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.956004 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.956026 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.956069 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.960310 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.961000 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.961013 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.961015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.961185 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.961557 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.961565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.961568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.961933 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.962149 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.962543 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.962815 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":514,"thread_start_us":381,"threads_started":1}
I20260620 18:35:11.966013 22909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:11.967728 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.966912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41800)
0620 18:35:11.967192 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.967458 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 18:35:11.967996 22859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:11.969861 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.969380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41810)
0620 18:35:11.969527 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.969671 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
W20260620 18:35:11.970255 22909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:11.971669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41539
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:12.019640 22909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 1
I20260620 18:35:12.020947 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.020395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41826)
0620 18:35:12.020639 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.020755 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:35:12.022302 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.021639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41832)
0620 18:35:12.021943 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.022122 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260620 18:35:12.022585 22909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.090878 22909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 2
I20260620 18:35:12.092229 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.091697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41840)
0620 18:35:12.091912 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.092029 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:35:12.093422 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.092898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41852)
0620 18:35:12.093189 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.093280 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:35:12.093713 22909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.112272 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } attempt: 5
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:12.127458 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.113097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40324)
0620 18:35:12.113307 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.113335 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.113437 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.113855 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.113859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.114265 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.114445 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.114455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.115910 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.115914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.116836 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.116850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.117863 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.117867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.118051 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.118771 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.118800 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.121012 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.124919 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.124924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.124927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.125354 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.125817 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.125821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.125823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.125981 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.126713 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.126716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.126854 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.127003 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.127214 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:35:12.127770 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.113177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.0.97.61:45481)
0620 18:35:12.113384 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.113390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.113415 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:35:12.113573 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.113581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.113680 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.113826 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.114611 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.115755 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.117006 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.117687 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.117841 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.121223 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.121230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.121234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.121276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.124758 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.125491 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.125500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.125505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.125681 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.126105 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.126113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.126117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.126573 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.126758 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.127154 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.127405 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:35:12.128907 22835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:12.129931 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.129562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41862)
0620 18:35:12.129733 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.129801 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:35:12.130823 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.130514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41866)
0620 18:35:12.130660 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.130733 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:35:12.131075 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.182346 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 1
I20260620 18:35:12.183652 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.183132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41882)
0620 18:35:12.183331 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.183427 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:35:12.184823 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.184467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41890)
0620 18:35:12.184634 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.184721 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:35:12.184988 22909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 3
W20260620 18:35:12.185149 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.186090 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.185501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41902)
0620 18:35:12.185748 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.185866 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:35:12.187306 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.186733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41904)
0620 18:35:12.187008 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.187137 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260620 18:35:12.187594 22909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.253412 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 2
I20260620 18:35:12.254570 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.254084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41912)
0620 18:35:12.254312 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.254405 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:35:12.255666 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.255289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41926)
0620 18:35:12.255437 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.255558 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:35:12.255963 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
W20260620 18:35:12.309484 22916 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:12.309837 22916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:12.309966 22916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:12.310086 22916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:12.317302 22916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:12.317390 22916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:12.317479 22916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:12.317581 22916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:12.317687 22916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:12.327631 22916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35801,127.0.97.61:45481,127.0.97.60:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:12.329018 22916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:12.330400 22916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:35:12.330834 22909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 4
I20260620 18:35:12.332166 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.331618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41930)
0620 18:35:12.331870 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.331973 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:35:12.333230 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.332829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41932)
0620 18:35:12.333004 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.333099 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:35:12.333508 22909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
W20260620 18:35:12.343719 22924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:12.343663 22921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:12.343691 22922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:12.349318 22916 server_base.cc:1034] running on GCE node
I20260620 18:35:12.350219 22916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:12.351766 22916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:12.353018 22916 hybrid_clock.cc:648] HybridClock initialized: now 1781980512352940 us; error 57 us; skew 500 ppm
I20260620 18:35:12.353369 22916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:12.356835 22916 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 18:35:12.358423 22916 webserver.cc:460] Webserver started at http://127.0.97.60:39645/ using document root <none> and password file <none>
I20260620 18:35:12.359169 22916 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:12.359304 22916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:12.359700 22916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:12.362221 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 3
I20260620 18:35:12.362916 22916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data/instance:
uuid: "4aaac71381d041c28a1a8b492899d43c"
format_stamp: "Formatted at 2026-06-20 18:35:12 on dist-test-slave-mcnm"
server_key: "72bbf1dc4c5ee77cfbd888a835f23eff"
server_key_iv: "c17023a4de2b18eb828e09bcf5fd43cb"
server_key_version: "encryptionkey@0"
I20260620 18:35:12.363399 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.362890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41938)
0620 18:35:12.363105 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.363195 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:35:12.364012 22916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal/instance:
uuid: "4aaac71381d041c28a1a8b492899d43c"
format_stamp: "Formatted at 2026-06-20 18:35:12 on dist-test-slave-mcnm"
server_key: "72bbf1dc4c5ee77cfbd888a835f23eff"
server_key_iv: "c17023a4de2b18eb828e09bcf5fd43cb"
server_key_version: "encryptionkey@0"
I20260620 18:35:12.364516 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.364060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41948)
0620 18:35:12.364339 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.364409 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260620 18:35:12.364845 22835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41539: connect: Connection refused (error 111)
I20260620 18:35:12.369508 22916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:12.373250 22931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:12.374639 22916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:35:12.374821 22916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "4aaac71381d041c28a1a8b492899d43c"
format_stamp: "Formatted at 2026-06-20 18:35:12 on dist-test-slave-mcnm"
server_key: "72bbf1dc4c5ee77cfbd888a835f23eff"
server_key_iv: "c17023a4de2b18eb828e09bcf5fd43cb"
server_key_version: "encryptionkey@0"
I20260620 18:35:12.375011 22916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:12.393680 22916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:12.398496 22916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:12.398808 22916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:12.418027 22916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41539
I20260620 18:35:12.418048 22982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41539 every 8 connection(s)
I20260620 18:35:12.419351 22916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
I20260620 18:35:12.423007 22983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:12.427789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 22916
I20260620 18:35:12.427970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980152255989-388-0/minicluster-data/master-2/wal/instance
I20260620 18:35:12.428514 388 external_mini_cluster.cc:1428] Setting key 5891dbf66674cd56d1f2a2821fd814d5
I20260620 18:35:12.430336 22983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:35:12.448657 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.433248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.0.97.62:35801)
0620 18:35:12.433421 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.433425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.434169 (+ 744us) server_negotiation.cc:407] Connection header received
0620 18:35:12.434489 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.434495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.434596 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.434711 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.436022 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.436729 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.438207 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.438645 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.438777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.442606 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.442611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.442614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.442648 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.445642 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.446363 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.446369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.446373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.446513 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.446974 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.446979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.446983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.447255 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.447431 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.448040 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.448284 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:35:12.449625 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.430528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.0.97.60:41539)
0620 18:35:12.431505 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.431554 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.431608 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:35:12.433764 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.433797 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.434258 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.434535 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.436039 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.437200 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.438477 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.439000 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.439144 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.443235 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.443243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.443255 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.443286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.445910 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.446574 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.446580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.446582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.446723 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.447092 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.447098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.447101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.447681 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.447991 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.448302 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.448597 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":729,"thread_start_us":457,"threads_started":1}
I20260620 18:35:12.449663 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.433301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35801 (local address 127.0.0.1:44354)
0620 18:35:12.434000 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.434035 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.434254 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.434772 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.434778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.435414 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.435846 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.435868 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.436896 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.436901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.438063 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.438075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.438902 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.438909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.439092 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.439876 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.439908 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.442296 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.445778 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.445791 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.445806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.446240 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.446672 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.446677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.446681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.446857 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.447390 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.447400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.447681 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.447947 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.448453 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":365,"threads_started":1}
I20260620 18:35:12.451305 22983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:35:12.466753 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.452225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45481 (local address 127.0.0.1:40326)
0620 18:35:12.452553 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.452572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.452698 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.453258 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.453263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.453642 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.453777 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.453785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.454787 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.454795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.455831 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.455842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.456527 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.456532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.456718 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.457999 (+ 1281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.458028 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.460686 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.464001 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.464006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.464010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.464539 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.465059 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.465064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.465068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.465231 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.465780 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.465784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.465937 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.466089 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.466371 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260620 18:35:12.466787 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.452211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.97.61:45481)
0620 18:35:12.452415 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.452423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.452968 (+ 545us) server_negotiation.cc:407] Connection header received
0620 18:35:12.453029 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.453033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.453094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.453243 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.453931 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.454641 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.455965 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.456384 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.456519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.460937 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.460942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.460946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.461000 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.463876 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.464703 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.464707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.464709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.464925 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.465346 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.465352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.465356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.465659 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.465816 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.466178 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.466380 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:35:12.467818 22983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:12.468849 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.452880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.97.62:35801)
0620 18:35:12.453063 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.453071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.453089 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:12.453144 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.453149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.453209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.453308 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.454498 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.455315 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.456632 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.457168 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.457302 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.462907 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.462913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.462916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.462956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.465757 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.466394 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.466401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.466404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.466524 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.467081 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.467085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.467088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.467400 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.467664 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.468263 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.468469 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:12.481352 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.468499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41970)
0620 18:35:12.468661 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.468680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.468775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.469300 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.469305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.469691 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.469852 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.469863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.471164 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.471169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.472180 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.472192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.472860 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.472866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.473077 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.473623 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.473639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.475686 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.478715 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.478722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.478726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.479102 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.479495 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.479498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.479500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.479682 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.480366 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.480370 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.480503 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.480641 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.480844 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:35:12.481387 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.468742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.97.60:41539)
0620 18:35:12.468977 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.468986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.469005 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:12.469051 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.469055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.469136 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.469218 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.470094 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.471047 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.472325 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.472725 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.472855 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.475854 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.475858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.475861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.475886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.478590 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.479216 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.479223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.479226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.479376 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.479789 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.479794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.479796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.480193 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.480344 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.480788 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.481005 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 18:35:12.485150 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.470877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.97.61:45481)
0620 18:35:12.471035 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.471039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.471052 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:35:12.471087 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.471090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.471178 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.471276 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.472469 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.473162 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.474337 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.474957 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.475076 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.479442 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.479451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.479456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.479492 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.482437 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.483209 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.483214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.483217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.483388 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.483829 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.483833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.483836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.484172 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.484430 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.484602 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.484808 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:12.487566 22983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: Bootstrap starting.
I20260620 18:35:12.489893 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:12.490958 22983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:12.492012 22983 log.cc:824] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:12.495276 22983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: No bootstrap required, opened a new log
I20260620 18:35:12.492550 22810 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "450664b3449045319063e68c94b4bbf4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44362
I20260620 18:35:12.505999 22983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.506417 22983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaac71381d041c28a1a8b492899d43c, State: Initialized, Role: FOLLOWER
I20260620 18:35:12.506937 22983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.507933 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.508266 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.509173 22983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:35:12.519316 23000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:35:12.519464 23000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:35:12.516619 22916 master_runner.cc:186] Error getting master registration for 127.0.97.62:35801: OK, instance_id { permanent_uuid: "450664b3449045319063e68c94b4bbf4" instance_seqno: 1781980511395637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:12.525878 22916 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:35:12.542085 22835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 4
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:35:12.557495 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.542778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:41986)
0620 18:35:12.542974 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.543000 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.543099 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.543472 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.543477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.543912 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.544076 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.544085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.545485 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.545491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.547006 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.547022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.548030 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.548038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.548241 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.548901 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.548928 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.550962 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.554812 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.554817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.554822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.555183 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.555635 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.555640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.555643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.555805 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.556511 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.556516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.556667 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.556815 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.557105 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:35:12.557524 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.542864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.0.97.60:41539)
0620 18:35:12.543085 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.543092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.543114 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:35:12.543183 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.543188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.543266 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.543373 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.544213 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.545339 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.547180 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.547883 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.548024 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.551153 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.551158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.551162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.551201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.554685 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.555313 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.555321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.555325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.555489 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.555918 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.555925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.555929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.556387 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.556543 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.556962 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.557182 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:35:12.562719 22835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: Bootstrap starting.
I20260620 18:35:12.566041 22835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:12.567013 22835 log.cc:824] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:12.570039 22835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: No bootstrap required, opened a new log
I20260620 18:35:12.575248 22835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.575618 22835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450664b3449045319063e68c94b4bbf4, State: Initialized, Role: FOLLOWER
I20260620 18:35:12.576030 22835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.576924 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.577288 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.578130 22835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:12.578294 22810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:35:12.578553 22810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:12.582195 22810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.585014 22810 leader_election.cc:288] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [CANDIDATE]: Term 1 election: Requested vote from peers d15843def5ec49b880b9eb425596af03 (127.0.97.61:45481), 4aaac71381d041c28a1a8b492899d43c (127.0.97.60:41539)
I20260620 18:35:12.585352 22958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "450664b3449045319063e68c94b4bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4aaac71381d041c28a1a8b492899d43c"
I20260620 18:35:12.585824 22958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:12.589435 22958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450664b3449045319063e68c94b4bbf4 in term 1.
I20260620 18:35:12.590159 22785 leader_election.cc:302] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450664b3449045319063e68c94b4bbf4, 4aaac71381d041c28a1a8b492899d43c; no voters:
I20260620 18:35:12.590619 23003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:12.600183 23003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [term 1 LEADER]: Becoming Leader. State: Replica: 450664b3449045319063e68c94b4bbf4, State: Running, Role: LEADER
I20260620 18:35:12.600857 23003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.605485 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450664b3449045319063e68c94b4bbf4. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.605971 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:12.609193 23014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:12.618686 23014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:12.632738 22909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } attempt: 5
W20260620 18:35:12.634686 22789 tablet.cc:2369] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:12.644734 22958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [term 1 FOLLOWER]: Refusing update from remote peer 450664b3449045319063e68c94b4bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 20 18:35:12 dist-test-slave-mcnm krb5kdc[22755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:35:12.645887 23012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:35:12.651042 22937 tablet.cc:2369] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:12.657442 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.633470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41539 (local address 127.0.0.1:42000)
0620 18:35:12.633635 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:12.633656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:12.633778 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:12.635621 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:12.635629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:12.636275 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:12.636550 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.636565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.638431 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.638439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.640297 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:12.640318 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.641834 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.641843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:12.642124 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.642896 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:12.642929 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:12.646552 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:12.653906 (+ 7354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.653914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.653921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.654580 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.655132 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:12.655140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:12.655147 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:12.655359 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:12.656223 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:12.656230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:12.656438 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 18:35:12.656614 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:35:12.656992 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:35:12.658617 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.633598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.97.60:41539)
0620 18:35:12.633863 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.633872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.633896 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:12.633977 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.633985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.634082 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.634212 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.636811 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.638232 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.640520 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.641400 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.641540 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.646826 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.646835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.646843 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.646896 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.653681 (+ 6785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.654766 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.654776 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.654782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.654978 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.655508 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.655537 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.655543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.655981 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.656151 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.657837 (+ 1686us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.658143 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 18:35:12.666157 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: Bootstrap starting.
I20260620 18:35:12.667567 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450664b3449045319063e68c94b4bbf4. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.671736 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.673589 22909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:12.676905 22909 log.cc:824] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:12.678005 23019 mvcc.cc:205] Tried to move back new op lower bound from 7298992179751305216 to 7298992179618988032. Current Snapshot: MvccSnapshot[applied={T|T < 7298992179751305216}]
I20260620 18:35:12.680510 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.680850 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:12.685357 23014 catalog_manager.cc:1347] Generated new cluster ID: 4311c89691d44af89cbe4d06ea1253d5
I20260620 18:35:12.685472 23014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:12.695478 23022 mvcc.cc:205] Tried to move back new op lower bound from 7298992179751305216 to 7298992179618988032. Current Snapshot: MvccSnapshot[applied={T|T < 7298992179751305216}]
I20260620 18:35:12.740360 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.740665 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.747588 23014 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:12.749428 23014 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:12.752370 22768 master_runner.cc:186] Error getting master registration for 127.0.97.61:45481: OK, instance_id { permanent_uuid: "d15843def5ec49b880b9eb425596af03" instance_seqno: 1781980511897652 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:12.770078 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: No bootstrap required, opened a new log
I20260620 18:35:12.775753 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.699815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.97.61:45481)
0620 18:35:12.700085 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.700092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.732047 (+ 31955us) server_negotiation.cc:407] Connection header received
0620 18:35:12.732338 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.732344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.732430 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.732501 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.734601 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.735788 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.747006 (+ 11218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.747753 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.747866 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.751475 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.751482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.751487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.751552 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.772057 (+ 20505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.773040 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.773046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.773050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.773274 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.773756 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.773761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.773765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.774169 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.774398 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.775030 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.775324 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:35:12.782734 22768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:35:12.788465 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.707332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.97.60:41539)
0620 18:35:12.715744 (+ 8412us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.715750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.745237 (+ 29487us) server_negotiation.cc:407] Connection header received
0620 18:35:12.753599 (+ 8362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.753604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.753691 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.753791 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.755636 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.756781 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.760442 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.761169 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.761318 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.764863 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.764872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.764878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.764927 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.782426 (+ 17499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.785270 (+ 2844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.785276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.785282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.785450 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.785997 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.786004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.786010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.786488 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.786745 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.787792 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.788040 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8254}
I20260620 18:35:12.788410 23014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4: Generated new TSK 0
I20260620 18:35:12.788982 22909 raft_consensus.cc:348] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.789417 22909 raft_consensus.cc:729] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15843def5ec49b880b9eb425596af03, State: Initialized, Role: FOLLOWER
I20260620 18:35:12.789480 23014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:12.789956 22909 consensus_queue.cc:260] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } }
I20260620 18:35:12.794960 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:12.680755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.0.97.62:35801)
0620 18:35:12.680964 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.680970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.728692 (+ 47722us) server_negotiation.cc:407] Connection header received
0620 18:35:12.729475 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.729481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.729580 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.729657 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.747955 (+ 18298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.749076 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.760408 (+ 11332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:12.761184 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:12.761323 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:12.783044 (+ 21721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:12.783052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:12.783057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:12.783100 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:12.788189 (+ 5089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.792472 (+ 4283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.792480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.792484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.792656 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:12.793123 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:12.793128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:12.793132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:12.793474 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:12.793725 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:12.794264 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 18:35:12.794560 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":91}
I20260620 18:35:12.795342 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.797857 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.797796 22884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "450664b3449045319063e68c94b4bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d15843def5ec49b880b9eb425596af03"
I20260620 18:35:12.797727 22909 sys_catalog.cc:564] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:12.798270 22884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:12.802363 22884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450664b3449045319063e68c94b4bbf4 in term 1.
I20260620 18:35:12.803144 22883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [term 1 FOLLOWER]: Refusing update from remote peer 450664b3449045319063e68c94b4bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:12.804847 23003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:12.834627 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450664b3449045319063e68c94b4bbf4. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.834976 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.836234 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:12.836597 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:12.935467 23058 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 18:35:12.945092 22839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.911994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.0.97.62:35801)
0620 18:35:12.912230 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.912237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.941051 (+ 28814us) server_negotiation.cc:407] Connection header received
0620 18:35:12.942977 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.942983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.943066 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.943170 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.944840 (+ 1670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44380: BlockingRecv error: recv got EOF from 127.0.0.1:44380 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:35:12.949216 22986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.925836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.0.97.60:41539)
0620 18:35:12.926042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.926053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.947153 (+ 21100us) server_negotiation.cc:407] Connection header received
0620 18:35:12.947529 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.947535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.947613 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.947683 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.948975 (+ 1292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42016: BlockingRecv error: recv got EOF from 127.0.0.1:42016 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 18:35:12.950560 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.918275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.97.61:45481)
0620 18:35:12.918461 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.918479 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.946643 (+ 28164us) server_negotiation.cc:407] Connection header received
0620 18:35:12.947913 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.947919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.947995 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.948067 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.950341 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40344: BlockingRecv error: recv got EOF from 127.0.0.1:40344 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 18:35:12.965552 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.963261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.0.97.61:45481)
0620 18:35:12.963466 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.963472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.963980 (+ 508us) server_negotiation.cc:407] Connection header received
0620 18:35:12.964191 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.964197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.964324 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.964439 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.965257 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 18:35:12.980588 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.978289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.97.61:45481)
0620 18:35:12.978527 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.978535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.978898 (+ 363us) server_negotiation.cc:407] Connection header received
0620 18:35:12.979204 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.979211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.979309 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.979442 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.980371 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40372: BlockingRecv error: recv got EOF from 127.0.0.1:40372 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:35:12.993677 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:12.987438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.0.97.61:45481)
0620 18:35:12.987680 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:12.987687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:12.992112 (+ 4425us) server_negotiation.cc:407] Connection header received
0620 18:35:12.992304 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:12.992311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:12.992423 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:12.992541 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:12.993436 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40386: BlockingRecv error: recv got EOF from 127.0.0.1:40386 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:35:13.003592 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.001317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.61:45481)
0620 18:35:13.001529 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.001538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.002053 (+ 515us) server_negotiation.cc:407] Connection header received
0620 18:35:13.002257 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.002263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.002365 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.002486 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.003309 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40398: BlockingRecv error: recv got EOF from 127.0.0.1:40398 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 18:35:13.018874 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.016597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.0.97.61:45481)
0620 18:35:13.016797 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.016804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.017363 (+ 559us) server_negotiation.cc:407] Connection header received
0620 18:35:13.017564 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.017571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.017670 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.017788 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.018631 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40410: BlockingRecv error: recv got EOF from 127.0.0.1:40410 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:35:13.044339 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.041843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.97.61:45481)
0620 18:35:13.042055 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.042062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.042627 (+ 565us) server_negotiation.cc:407] Connection header received
0620 18:35:13.042858 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.042866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.042969 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.043097 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.044067 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40416: BlockingRecv error: recv got EOF from 127.0.0.1:40416 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:35:13.082974 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.080585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.97.61:45481)
0620 18:35:13.080785 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.080791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.081361 (+ 570us) server_negotiation.cc:407] Connection header received
0620 18:35:13.081569 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.081574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.081680 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.081804 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.082710 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40420: BlockingRecv error: recv got EOF from 127.0.0.1:40420 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:35:13.152770 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.150643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.0.97.61:45481)
0620 18:35:13.150914 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.150920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.151246 (+ 326us) server_negotiation.cc:407] Connection header received
0620 18:35:13.151455 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.151460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.151588 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.151688 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.152543 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40422: BlockingRecv error: recv got EOF from 127.0.0.1:40422 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 18:35:13.207283 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "450664b3449045319063e68c94b4bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450664b3449045319063e68c94b4bbf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35801 } } peers { permanent_uuid: "d15843def5ec49b880b9eb425596af03" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45481 } } peers { permanent_uuid: "4aaac71381d041c28a1a8b492899d43c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41539 } } }
I20260620 18:35:13.207732 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450664b3449045319063e68c94b4bbf4 [sys.catalog]: This master's current role is: LEADER
W20260620 18:35:13.286427 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.284292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.97.61:45481)
0620 18:35:13.284638 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.284645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.284828 (+ 183us) server_negotiation.cc:407] Connection header received
0620 18:35:13.285104 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.285109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.285236 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.285347 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.286169 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40434: BlockingRecv error: recv got EOF from 127.0.0.1:40434 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:35:13.521497 23000 catalog_manager.cc:1259] Loaded cluster ID: 4311c89691d44af89cbe4d06ea1253d5
I20260620 18:35:13.521618 23000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: loading cluster ID for follower catalog manager: success
I20260620 18:35:13.524423 23000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: acquiring CA information for follower catalog manager: success
I20260620 18:35:13.526280 23000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4aaac71381d041c28a1a8b492899d43c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:35:13.547906 22914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:13.546208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.0.97.61:45481)
0620 18:35:13.546380 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.546384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.546620 (+ 236us) server_negotiation.cc:407] Connection header received
0620 18:35:13.546780 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.546784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.546858 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.546953 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.547736 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40440: BlockingRecv error: recv got EOF from 127.0.0.1:40440 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:35:13.939986 23058 catalog_manager.cc:1259] Loaded cluster ID: 4311c89691d44af89cbe4d06ea1253d5
I20260620 18:35:13.940126 23058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: loading cluster ID for follower catalog manager: success
I20260620 18:35:13.942767 23058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: acquiring CA information for follower catalog manager: success
I20260620 18:35:13.944692 23058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d15843def5ec49b880b9eb425596af03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:35:14.077728 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.070551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.0.97.61:45481)
0620 18:35:14.071218 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.071226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.071672 (+ 446us) server_negotiation.cc:407] Connection header received
0620 18:35:14.071753 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.071757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.071900 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.072013 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:14.072509 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.073334 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.075269 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.075751 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.075849 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.076465 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:14.077061 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:14.077190 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.077306 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.077374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":314,"threads_started":1}
I20260620 18:35:14.090783 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.084068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.97.60:41539)
0620 18:35:14.084608 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.084614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.084853 (+ 239us) server_negotiation.cc:407] Connection header received
0620 18:35:14.085072 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.085077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.085314 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.085472 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:14.085908 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.086611 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.088515 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.088915 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.089010 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.089601 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:14.090110 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:14.090224 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.090359 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.090422 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260620 18:35:14.106158 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.097269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.0.97.61:45481)
0620 18:35:14.098060 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.098067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.099199 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 18:35:14.099468 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.099474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.099607 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.099687 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:14.100528 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.101724 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.103932 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.104613 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.104725 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.105333 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:14.105466 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:14.105524 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.105752 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.105845 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638}
I20260620 18:35:14.108645 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.098223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.97.62:35801)
0620 18:35:14.098895 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.098901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.098918 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:35:14.099046 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.099055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.099369 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.100365 (+ 996us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:14.100579 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.101967 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.104931 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.105800 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.106575 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.106807 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:14.107373 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:14.108212 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.108265 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.108350 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":380,"threads_started":1}
I20260620 18:35:14.113920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22768
I20260620 18:35:14.114926 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.098543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.97.60:41539)
0620 18:35:14.098734 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.098742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.108176 (+ 9434us) server_negotiation.cc:407] Connection header received
0620 18:35:14.108475 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.108484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.108597 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.108672 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:14.109460 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.110490 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.112783 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.113378 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.113526 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.114014 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:14.114145 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:14.114234 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.114576 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.114654 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:35:14.115029 22830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:14.227789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22768
I20260620 18:35:14.249063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22842
I20260620 18:35:14.250283 22904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:14.367122 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22842
I20260620 18:35:14.386859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 22916
I20260620 18:35:14.388077 22978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:14.497066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 22916
2026-06-20T18:35:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4941 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:14 dist-test-slave-mcnm krb5kdc[23136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:14 dist-test-slave-mcnm krb5kdc[23136](info): set up 2 sockets
Jun 20 18:35:14 dist-test-slave-mcnm krb5kdc[23136](info): commencing operation
krb5kdc: starting...
W20260620 18:35:15.632236 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/test-user.keytab.
Jun 20 18:35:15 dist-test-slave-mcnm krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 18:35:15.895149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33241
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:16.228856 23149 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:16.229187 23149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:16.229270 23149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:16.229357 23149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:16.236719 23149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:16.236832 23149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:16.236891 23149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:16.236935 23149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:16.236999 23149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:16.247066 23149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:16.248435 23149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:16.250051 23149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:16.262238 23154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:16.262193 23155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:16.262147 23157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:16.263244 23149 server_base.cc:1034] running on GCE node
I20260620 18:35:16.264168 23149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:16.265681 23149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:16.266950 23149 hybrid_clock.cc:648] HybridClock initialized: now 1781980516266884 us; error 39 us; skew 500 ppm
I20260620 18:35:16.267300 23149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:16 dist-test-slave-mcnm krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:16.270582 23149 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 18:35:16.272230 23149 webserver.cc:460] Webserver started at http://127.0.97.62:39391/ using document root <none> and password file <none>
I20260620 18:35:16.272955 23149 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:16.273082 23149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:16.273401 23149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:16.276523 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data/instance:
uuid: "24c84a2161c44804a97d497702fe0493"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "60c511091eb831e72879632c6ce23107"
server_key_iv: "06dd1d1919fb2f0473e30e76e163628d"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.277400 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance:
uuid: "24c84a2161c44804a97d497702fe0493"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "60c511091eb831e72879632c6ce23107"
server_key_iv: "06dd1d1919fb2f0473e30e76e163628d"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.282560 23149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:16.286139 23164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:16.287447 23149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:16.287695 23149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal
uuid: "24c84a2161c44804a97d497702fe0493"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "60c511091eb831e72879632c6ce23107"
server_key_iv: "06dd1d1919fb2f0473e30e76e163628d"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.287922 23149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:16.311141 23149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:16.315495 23149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:16.315788 23149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:16.334156 23149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33241
I20260620 18:35:16.334173 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33241 every 8 connection(s)
I20260620 18:35:16.335484 23149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:16.339228 23216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:16.340883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 23149
I20260620 18:35:16.341063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-0/wal/instance
I20260620 18:35:16.341516 388 external_mini_cluster.cc:1428] Setting key 4aef3b2334921bcd0253490646c81b2d
I20260620 18:35:16.345786 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:16 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 18:35:16.372527 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.0.97.62:33241)
0620 18:35:16.350764 (+ 1847us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:16.350771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:16.350811 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:35:16.353205 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:16.353211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:16.353478 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:16.353636 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:16.355294 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.356609 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.358897 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.359803 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.359946 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.364263 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:16.364271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:16.364284 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:16.364316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:16.368002 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.368804 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.368811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.368813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.368987 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.369468 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.369472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.369474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.370211 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:16.370413 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:16.371091 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 18:35:16.371373 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"spinlock_wait_cycles":22784,"thread_start_us":1450,"threads_started":1}
I20260620 18:35:16.372555 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33241 (local address 127.0.0.1:43532)
0620 18:35:16.349862 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.349924 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:16.352804 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:16.353740 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:16.353750 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:16.354560 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:16.355077 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.355104 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.356772 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.356780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.358725 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.358743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.360068 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.360078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.360302 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.361085 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:16.361121 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:16.364007 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:16.368138 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.368149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.368162 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.368683 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.369171 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.369179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.369185 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.369368 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.370391 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:16.370399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:16.370687 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 18:35:16.370979 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 18:35:16.371632 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":60,"thread_start_us":387,"threads_started":1}
I20260620 18:35:16.375483 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:16.376546 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.376191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52544)
0620 18:35:16.376347 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.376440 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:35:16.376866 23165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
I20260620 18:35:16.378280 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.377963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52548)
0620 18:35:16.378092 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.378165 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:16.378506 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:32957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 18:35:16.403808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:32957
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:16.427886 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } attempt: 1
I20260620 18:35:16.429176 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.428711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52550)
0620 18:35:16.428940 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.429049 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:35:16.430224 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.429908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52564)
0620 18:35:16.430041 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.430108 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:35:16.430474 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:32957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
I20260620 18:35:16.498724 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } attempt: 2
I20260620 18:35:16.499977 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.499468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52572)
0620 18:35:16.499740 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.499835 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:35:16.501076 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.500706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52584)
0620 18:35:16.500871 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.500953 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 18:35:16.501390 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:32957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
I20260620 18:35:16.592677 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } attempt: 3
I20260620 18:35:16.593845 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.593402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52596)
0620 18:35:16.593598 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.593691 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:35:16.594995 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.594650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52598)
0620 18:35:16.594807 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.594873 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 18:35:16.595279 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:32957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
W20260620 18:35:16.737367 23223 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:16.737731 23223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:16.737845 23223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:16.737984 23223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:35:16.738621 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } attempt: 4
I20260620 18:35:16.739992 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.739360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52606)
0620 18:35:16.739760 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.739856 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260620 18:35:16.741086 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.740696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52608)
0620 18:35:16.740861 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.740960 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:35:16.741322 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:32957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:32957: connect: Connection refused (error 111)
W20260620 18:35:16.745748 23223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:16.745867 23223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:16.745990 23223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:16.746074 23223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:16.746186 23223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:16.756413 23223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:16.757768 23223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:16.759177 23223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:16.775436 23229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:16.775490 23231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:16.775918 23228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:16.776437 23223 server_base.cc:1034] running on GCE node
I20260620 18:35:16.777367 23223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:16.778823 23223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:16.780083 23223 hybrid_clock.cc:648] HybridClock initialized: now 1781980516779993 us; error 57 us; skew 500 ppm
I20260620 18:35:16.780447 23223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:16 dist-test-slave-mcnm krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:16.783720 23223 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 18:35:16.785174 23223 webserver.cc:460] Webserver started at http://127.0.97.61:40245/ using document root <none> and password file <none>
I20260620 18:35:16.785928 23223 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:16.786012 23223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:16.786266 23223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:16.789178 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data/instance:
uuid: "88a72b4bfa384bc9b9a1a0058785b540"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "7edcc7fb8f6326c31431910c20e95923"
server_key_iv: "483a84f2d7f4ba700d8cd6dd56f3abf6"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.789886 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal/instance:
uuid: "88a72b4bfa384bc9b9a1a0058785b540"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "7edcc7fb8f6326c31431910c20e95923"
server_key_iv: "483a84f2d7f4ba700d8cd6dd56f3abf6"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.795401 23223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:35:16.799058 23238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:16.800522 23223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:16.800697 23223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal
uuid: "88a72b4bfa384bc9b9a1a0058785b540"
format_stamp: "Formatted at 2026-06-20 18:35:16 on dist-test-slave-mcnm"
server_key: "7edcc7fb8f6326c31431910c20e95923"
server_key_iv: "483a84f2d7f4ba700d8cd6dd56f3abf6"
server_key_version: "encryptionkey@0"
I20260620 18:35:16.800923 23223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:16.817303 23223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:16.821967 23223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:16.822252 23223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:16.841861 23223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:32957
I20260620 18:35:16.841881 23289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:32957 every 8 connection(s)
I20260620 18:35:16.843209 23223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/data/info.pb
I20260620 18:35:16.846849 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:16.850315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 23223
I20260620 18:35:16.850488 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-1/wal/instance
I20260620 18:35:16.851037 388 external_mini_cluster.cc:1428] Setting key 54f6edd1a5490ce93e1bbb260ac37309
I20260620 18:35:16.855686 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:16 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 18:35:16.879913 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.859309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.0.97.62:33241)
0620 18:35:16.859549 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:16.859557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:16.861732 (+ 2175us) server_negotiation.cc:407] Connection header received
0620 18:35:16.863538 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:16.863548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:16.863653 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:16.863780 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:16.865286 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.866451 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.868103 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.868854 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.868997 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.872644 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:16.872654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:16.872662 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:16.872711 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:16.876234 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.877268 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.877274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.877277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.877405 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.877947 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.877952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.877954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.878266 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:16.878385 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:16.879409 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0620 18:35:16.879648 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:35:16.880916 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.860104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33241 (local address 127.0.0.1:43536)
0620 18:35:16.861063 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.861117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:16.863226 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:16.863977 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:16.863987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:16.864540 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:16.865094 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.865111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.866626 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.866630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.867933 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.867945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.869098 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.869104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.869326 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.870070 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:16.870094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:16.872400 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:16.876389 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.876401 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.876424 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.877131 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.877521 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.877529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.877535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.877784 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.878409 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:16.878417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:16.878933 (+ 516us) client_negotiation.cc:769] Sending connection context
0620 18:35:16.879259 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 18:35:16.879795 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":430,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 18:35:16.882892 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:16 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:16.899791 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.883584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52616)
0620 18:35:16.883886 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.883903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:16.884004 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:16.885000 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:16.885007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:16.885500 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:16.885708 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.885853 (+ 145us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.887318 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.887326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.888902 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:16.888919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.889813 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.889822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:16.890038 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.890668 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:16.890696 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:16.892984 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:16.897283 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.897287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.897289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.897647 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.898081 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:16.898084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:16.898087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:16.898259 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:16.899001 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:16.899004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:16.899120 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 18:35:16.899244 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:35:16.899441 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 18:35:16.900439 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.883894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.0.97.61:32957)
0620 18:35:16.884582 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:16.884588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:16.884612 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:16.884655 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:16.884660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:16.884832 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:16.884947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:16.886050 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.887143 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.889053 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:16.889623 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:16.889729 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:16.893199 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:16.893214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:16.893242 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:16.893284 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:16.897138 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.897790 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.897801 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.897807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.897947 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:16.898400 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:16.898409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:16.898415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:16.898872 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:16.899073 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:16.899451 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 18:35:16.899766 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":394,"threads_started":1}
I20260620 18:35:16.902778 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:16.904116 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.903482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39278)
0620 18:35:16.903777 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.903929 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260620 18:35:16.904431 23242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:16.906361 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.905894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39282)
0620 18:35:16.906080 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.906192 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:35:16.906623 23290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 18:35:16.920300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
/tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33173
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:16.955989 23290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 1
I20260620 18:35:16.957301 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.956754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39284)
0620 18:35:16.956987 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.957097 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:35:16.958374 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:16.957987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39300)
0620 18:35:16.958157 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:16.958247 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:16.958815 23290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:17.027110 23290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 2
I20260620 18:35:17.028442 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.027917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39308)
0620 18:35:17.028158 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.028272 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 18:35:17.029596 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.029246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39310)
0620 18:35:17.029414 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.029483 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:35:17.029865 23290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:17.040565 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } attempt: 5
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:17.056023 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.041330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52624)
0620 18:35:17.041546 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.041568 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.041705 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.042081 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.042084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.042481 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.042651 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.042659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.044078 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.044082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.044978 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.044997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.045994 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.045999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.046165 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.046886 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.046905 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.049008 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.053307 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.053311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.053314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.053707 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.054124 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.054137 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.054141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.054284 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.055033 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.055037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.055187 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.055310 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.055745 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:35:17.056057 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.041528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.0.97.61:32957)
0620 18:35:17.041726 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.041733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.041756 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:17.041824 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.041829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.041928 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.042040 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.042802 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.043916 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.045148 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.045804 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.045939 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.049261 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.049271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.049279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.049329 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.053153 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.053829 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.053834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.053838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.053988 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.054396 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.054401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.054404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.054887 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.055091 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.055461 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.055735 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:35:17.057413 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:17.058446 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.058057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39318)
0620 18:35:17.058225 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.058313 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:35:17.059442 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.059140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39332)
0620 18:35:17.059264 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.059341 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:35:17.059700 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:17.110945 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 1
I20260620 18:35:17.112186 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.111709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39336)
0620 18:35:17.111931 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.112023 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:35:17.113241 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.112935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39340)
0620 18:35:17.113069 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.113137 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:35:17.113490 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:17.121109 23290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 3
I20260620 18:35:17.122377 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.121835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39356)
0620 18:35:17.122110 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.122208 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 18:35:17.123747 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.123145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39360)
0620 18:35:17.123443 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.123589 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 18:35:17.123963 23290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
I20260620 18:35:17.181747 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 2
I20260620 18:35:17.183141 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.182562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39370)
0620 18:35:17.182873 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.182977 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:35:17.184195 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.183855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39386)
0620 18:35:17.184004 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.184087 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:35:17.184437 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
W20260620 18:35:17.263827 23297 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:35:17.264192 23297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:17.264317 23297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:17.264462 23297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:35:17.267187 23290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 4
I20260620 18:35:17.268325 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.267846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39392)
0620 18:35:17.268082 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.268165 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:35:17.269305 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.268993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39406)
0620 18:35:17.269147 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.269207 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:35:17.269500 23290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
W20260620 18:35:17.272156 23297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:17.272279 23297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:17.272390 23297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:17.272475 23297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:17.272581 23297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:17.282508 23297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33241,127.0.97.61:32957,127.0.97.60:33173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:17.283859 23297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:17.285259 23297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:35:17.290645 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 3
I20260620 18:35:17.291939 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.291297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39416)
0620 18:35:17.291652 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.291760 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 18:35:17.293038 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.292608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39420)
0620 18:35:17.292820 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.292917 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260620 18:35:17.293248 23216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33173: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33173: connect: Connection refused (error 111)
W20260620 18:35:17.297526 23305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:17.297586 23302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:17.297639 23303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:17.298379 23297 server_base.cc:1034] running on GCE node
I20260620 18:35:17.299237 23297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:17.300804 23297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:17.302088 23297 hybrid_clock.cc:648] HybridClock initialized: now 1781980517302023 us; error 54 us; skew 500 ppm
I20260620 18:35:17.302474 23297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:17.305851 23297 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 18:35:17.307439 23297 webserver.cc:460] Webserver started at http://127.0.97.60:41391/ using document root <none> and password file <none>
I20260620 18:35:17.308259 23297 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:17.308403 23297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:17.308765 23297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:17.312029 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data/instance:
uuid: "97e819ee17e54ff988c331eecc59c97f"
format_stamp: "Formatted at 2026-06-20 18:35:17 on dist-test-slave-mcnm"
server_key: "5c3a530e15e00742b6d51412cfc094c4"
server_key_iv: "adc4f512d6c1f1962109215c93da076e"
server_key_version: "encryptionkey@0"
I20260620 18:35:17.312886 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal/instance:
uuid: "97e819ee17e54ff988c331eecc59c97f"
format_stamp: "Formatted at 2026-06-20 18:35:17 on dist-test-slave-mcnm"
server_key: "5c3a530e15e00742b6d51412cfc094c4"
server_key_iv: "adc4f512d6c1f1962109215c93da076e"
server_key_version: "encryptionkey@0"
I20260620 18:35:17.318285 23297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:17.321825 23312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:17.323340 23297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:35:17.323618 23297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal
uuid: "97e819ee17e54ff988c331eecc59c97f"
format_stamp: "Formatted at 2026-06-20 18:35:17 on dist-test-slave-mcnm"
server_key: "5c3a530e15e00742b6d51412cfc094c4"
server_key_iv: "adc4f512d6c1f1962109215c93da076e"
server_key_version: "encryptionkey@0"
I20260620 18:35:17.323866 23297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:17.365746 23297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:17.370630 23297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:17.371022 23297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:17.392086 23297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33173
I20260620 18:35:17.392094 23363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33173 every 8 connection(s)
I20260620 18:35:17.393489 23297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/data/info.pb
I20260620 18:35:17.396898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu as pid 23297
I20260620 18:35:17.397104 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcfg4Mf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980152255989-388-0/minicluster-data/master-2/wal/instance
I20260620 18:35:17.397318 23364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:17.397665 388 external_mini_cluster.cc:1428] Setting key 761079243fca2d689cff3e38e5eabeee
I20260620 18:35:17.407476 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 18:35:17.424080 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.399423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.0.97.60:33173)
0620 18:35:17.400320 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.400332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.400378 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:35:17.402017 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.402038 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.402458 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.402814 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.404867 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.406281 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.408379 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.409137 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.409278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.415727 (+ 6449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.415739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.415754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.415813 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.419643 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.420621 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.420635 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.420641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.420853 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.421237 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.421242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.421245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.421769 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.422116 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.422578 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.422962 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":398,"threads_started":1}
I20260620 18:35:17.430663 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.410587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33241 (local address 127.0.0.1:43542)
0620 18:35:17.413635 (+ 3048us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.413688 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.413879 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.415269 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.415275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.416279 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.416614 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.416623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.418013 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.418019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.419847 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.419857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.420618 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.420625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.420950 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.421590 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.421607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.423857 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.428081 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.428091 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.428101 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.428662 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.429075 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.429083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.429087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.429277 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.429796 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.429806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.429994 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.430145 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.430367 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"thread_start_us":2521,"threads_started":1}
I20260620 18:35:17.430944 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.410619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.0.97.62:33241)
0620 18:35:17.414838 (+ 4219us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.414846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.414871 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:35:17.414945 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.414951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.415044 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.415185 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.416828 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.417843 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.419975 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.420425 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.420572 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.424064 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.424069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.424072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.424106 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.427852 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.428820 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.428825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.428828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.428961 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.429393 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.429397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.429399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.429649 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.429777 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.430303 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.430524 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4047,"thread_start_us":3910,"threads_started":1}
I20260620 18:35:17.432381 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:17.447079 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.433114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32957 (local address 127.0.0.1:52638)
0620 18:35:17.433355 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.433376 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.433477 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.433843 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.433849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.434371 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.434604 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.434617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.435761 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.435771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.437260 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.437272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.438317 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.438322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.438483 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.439081 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.439098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.441077 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.444723 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.444729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.444734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.445087 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.445473 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.445479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.445483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.445627 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.446155 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.446160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.446308 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.446442 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.446716 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 18:35:17.447110 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.433305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.97.61:32957)
0620 18:35:17.433520 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.433528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.433547 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:17.433608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.433614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.433695 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.433795 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.434855 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.435594 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.437410 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.438138 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.438286 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.441280 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.441289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.441296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.441341 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.444589 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.445211 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.445218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.445226 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.445363 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.445741 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.445752 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.445755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.446019 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.446183 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.446542 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.446742 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:35:17.448094 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:17.449759 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.427596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.0.97.62:33241)
0620 18:35:17.428215 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.428219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.428237 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:17.428274 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.428277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.428337 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.428418 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.429901 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.431084 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.432907 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.433559 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.433685 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.444024 (+ 10339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.444029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.444032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.444065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.446688 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.447447 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.447452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.447455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.447628 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.448083 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.448090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.448093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.448451 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.448694 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.449148 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.449395 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":297,"threads_started":1}
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 18:35:17.463147 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.449109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39434)
0620 18:35:17.449333 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.449349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.449473 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.449894 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.449898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.450248 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.450367 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.450375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.451324 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.451330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.452273 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.452285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.453329 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.453337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.453595 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.454207 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.454235 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.456546 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.460462 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.460468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.460472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.460829 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.461215 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.461224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.461229 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.461415 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.462031 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.462037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.462193 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.462364 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.462694 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 18:35:17.463218 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.449100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.97.60:33173)
0620 18:35:17.449276 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.449282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.449422 (+ 140us) server_negotiation.cc:407] Connection header received
0620 18:35:17.449587 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.449591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.449668 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.449730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.450495 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.451183 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.452412 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.453074 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.453235 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.456714 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.456720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.456724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.456770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.460336 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.460977 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.460988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.460991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.461087 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.461560 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.461564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.461567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.461883 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.462070 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.462457 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.462862 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":76}
I20260620 18:35:17.467941 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.451300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.97.61:32957)
0620 18:35:17.451446 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.451451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.451607 (+ 156us) server_negotiation.cc:407] Connection header received
0620 18:35:17.451792 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.451797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.451859 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.451954 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.453336 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.454535 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.456098 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.457011 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.457107 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.461942 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.461946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.461949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.461978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.465236 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.465915 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.465921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.465925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.466075 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.466461 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.466466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.466469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.466802 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.467087 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.467351 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.467612 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:35:17.469048 23364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: Bootstrap starting.
I20260620 18:35:17.470466 23216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 4
I20260620 18:35:17.472998 23364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:17.473412 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:17.474375 23364 log.cc:824] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:17.477613 23364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: No bootstrap required, opened a new log
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:35:17.476574 23191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "24c84a2161c44804a97d497702fe0493"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558
I20260620 18:35:17.486207 23364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.486625 23364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e819ee17e54ff988c331eecc59c97f, State: Initialized, Role: FOLLOWER
I20260620 18:35:17.487197 23364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.488204 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.488600 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.489161 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.471128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39446)
0620 18:35:17.471506 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.471549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.471658 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.472042 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.472047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.472609 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.472845 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.472858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.474311 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.474317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.476062 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.476086 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.477196 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.477202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.477496 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.478224 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.478250 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.481488 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.486046 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.486051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.486054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.486558 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.487154 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.487161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.487165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.487345 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.488026 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.488033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.488231 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.488380 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.488720 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260620 18:35:17.489341 23364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:17.490156 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.471311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.0.97.60:33173)
0620 18:35:17.471577 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.471587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.471607 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:35:17.471784 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.471790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.471883 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.472015 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.473016 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.474143 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.476309 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.477027 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.477244 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.481695 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.481708 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.481717 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.481759 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.485896 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.486855 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.486860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.486863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.487009 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.487477 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.487483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.487487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.487869 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.488099 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.489495 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.489742 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:35:17.497399 23216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: Bootstrap starting.
I20260620 18:35:17.501643 23216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:17.503007 23216 log.cc:824] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: Log is configured to *not* fsync() on all Append() calls
W20260620 18:35:17.504634 23383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:35:17.504786 23383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:35:17.507023 23216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: No bootstrap required, opened a new log
I20260620 18:35:17.502058 23297 master_runner.cc:186] Error getting master registration for 127.0.97.62:33241: OK, instance_id { permanent_uuid: "24c84a2161c44804a97d497702fe0493" instance_seqno: 1781980516328941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:17.512251 23297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:35:17.513818 23216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.514142 23216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c84a2161c44804a97d497702fe0493, State: Initialized, Role: FOLLOWER
I20260620 18:35:17.514581 23216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.515475 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.515946 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.516860 23216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:17.517081 23191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:35:17.517318 23191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:17.521215 23191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.523765 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c84a2161c44804a97d497702fe0493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "97e819ee17e54ff988c331eecc59c97f"
I20260620 18:35:17.524252 23339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:17.527733 23191 leader_election.cc:288] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [CANDIDATE]: Term 1 election: Requested vote from peers 88a72b4bfa384bc9b9a1a0058785b540 (127.0.97.61:32957), 97e819ee17e54ff988c331eecc59c97f (127.0.97.60:33173)
I20260620 18:35:17.528131 23339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c84a2161c44804a97d497702fe0493 in term 1.
I20260620 18:35:17.528839 23168 leader_election.cc:302] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c84a2161c44804a97d497702fe0493, 97e819ee17e54ff988c331eecc59c97f; no voters:
I20260620 18:35:17.529400 23386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:17.530313 23386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [term 1 LEADER]: Becoming Leader. State: Replica: 24c84a2161c44804a97d497702fe0493, State: Running, Role: LEADER
I20260620 18:35:17.530874 23386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.535146 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c84a2161c44804a97d497702fe0493. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.535707 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:17.565690 23399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:17.568703 23290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } attempt: 5
W20260620 18:35:17.568909 23170 tablet.cc:2369] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:17.595329 23399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:17.621984 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.535437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.97.60:33173)
0620 18:35:17.535646 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.535655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.557169 (+ 21514us) server_negotiation.cc:407] Connection header received
0620 18:35:17.558610 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.558617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.558736 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.558818 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.577516 (+ 18698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.578680 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.585830 (+ 7150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.586544 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.586653 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.607541 (+ 20888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.607549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.607554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.607598 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.611355 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.613370 (+ 2015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.613376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.613379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.613540 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.614735 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.614741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.614744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.615077 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.615335 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.621303 (+ 5968us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.621568 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:35:17.623253 23318 tablet.cc:2369] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:17.626339 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.535426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.97.62:33241)
0620 18:35:17.536287 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.536296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.555385 (+ 19089us) server_negotiation.cc:407] Connection header received
0620 18:35:17.556727 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.556734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.556834 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.556915 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.567700 (+ 10785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.569107 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.585324 (+ 16217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.586485 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.586710 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.611683 (+ 24973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.611691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.611696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.611751 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.617408 (+ 5657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.618539 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.618546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.618549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.618722 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.619188 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.619207 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.619211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.619561 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.619809 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.625670 (+ 5861us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.625927 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641}
I20260620 18:35:17.627421 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.554841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.97.61:32957)
0620 18:35:17.555027 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.555033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.560486 (+ 5453us) server_negotiation.cc:407] Connection header received
0620 18:35:17.561402 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.561408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.561497 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.561571 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.581012 (+ 19441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.582134 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.588668 (+ 6534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.589339 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.589444 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.602692 (+ 13248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.602699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.602704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.602753 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.606536 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.612721 (+ 6185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.612728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.612732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.612917 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.613997 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.614003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.614006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.614399 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.614641 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.626772 (+ 12131us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.627043 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:35:17 dist-test-slave-mcnm krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 18:35:17.643472 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.569301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33173 (local address 127.0.0.1:39464)
0620 18:35:17.569585 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:17.569608 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:17.569749 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:17.599991 (+ 30242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:17.599997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:17.600500 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:17.600730 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.600742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.616234 (+ 15492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.616241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.617706 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:17.617722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.629315 (+ 11593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.629321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:17.629564 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.630365 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:17.630388 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:17.633168 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:17.640604 (+ 7436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.640610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.640615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.641138 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.641625 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:17.641630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:17.641634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:17.641808 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:17.642471 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:17.642476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:17.642638 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 18:35:17.642781 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:35:17.643076 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 18:35:17.644675 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:17.569415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.0.97.60:33173)
0620 18:35:17.598008 (+ 28593us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.598015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.598039 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:17.598097 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.598102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.598186 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.600886 (+ 2700us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.601160 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.602457 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.620305 (+ 17848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:17.621033 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:17.629939 (+ 8906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:17.633435 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:17.633442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:17.633446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:17.633482 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:17.640459 (+ 6977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.641315 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.641321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.641325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.641496 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:17.641955 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:17.641960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:17.641963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:17.642330 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:17.643882 (+ 1552us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:17.644018 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 18:35:17.644250 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28429,"thread_start_us":27832,"threads_started":1}
I20260620 18:35:17.650322 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: Bootstrap starting.
I20260620 18:35:17.655447 23339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [term 1 FOLLOWER]: Refusing update from remote peer 24c84a2161c44804a97d497702fe0493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:17.656442 23391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:17.672051 23290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:17.680239 23290 log.cc:824] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:17.688791 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: No bootstrap required, opened a new log
I20260620 18:35:17.697064 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c84a2161c44804a97d497702fe0493. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.697415 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.701309 23408 mvcc.cc:205] Tried to move back new op lower bound from 7298992200290443264 to 7298992199811694592. Current Snapshot: MvccSnapshot[applied={T|T < 7298992200290443264}]
I20260620 18:35:17.710798 23411 mvcc.cc:205] Tried to move back new op lower bound from 7298992200290443264 to 7298992199811694592. Current Snapshot: MvccSnapshot[applied={T|T < 7298992200290443264}]
I20260620 18:35:17.699688 23149 master_runner.cc:186] Error getting master registration for 127.0.97.61:32957: OK, instance_id { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" instance_seqno: 1781980516836393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:17.722489 23149 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:35:17.729585 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.729934 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:17.721973 23290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.730530 23290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a72b4bfa384bc9b9a1a0058785b540, State: Initialized, Role: FOLLOWER
I20260620 18:35:17.731230 23290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } }
I20260620 18:35:17.732798 23416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.733927 23399 catalog_manager.cc:1347] Generated new cluster ID: f96f2a98634a4c89837c4916d67d636f
I20260620 18:35:17.734021 23399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:17.734333 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.734584 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.735473 23290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:17.735651 23265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c84a2161c44804a97d497702fe0493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "88a72b4bfa384bc9b9a1a0058785b540"
I20260620 18:35:17.736116 23265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:17.740007 23265 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c84a2161c44804a97d497702fe0493 in term 1.
I20260620 18:35:17.747151 23264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [term 1 FOLLOWER]: Refusing update from remote peer 24c84a2161c44804a97d497702fe0493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:17.749558 23391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:17.758889 23416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.782052 23416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.782104 23420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c84a2161c44804a97d497702fe0493. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.782554 23420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.782606 23416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:17.788756 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24c84a2161c44804a97d497702fe0493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c84a2161c44804a97d497702fe0493" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33241 } } peers { permanent_uuid: "88a72b4bfa384bc9b9a1a0058785b540" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32957 } } peers { permanent_uuid: "97e819ee17e54ff988c331eecc59c97f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33173 } } }
I20260620 18:35:17.789028 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:17.799324 23399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:17.803869 23399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:17.828363 23399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c84a2161c44804a97d497702fe0493: Generated new TSK 0
I20260620 18:35:17.833674 23399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:17.865258 23432 catalog_manager.cc:1259] Loaded cluster ID: f96f2a98634a4c89837c4916d67d636f
I20260620 18:35:17.865375 23432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: loading cluster ID for follower catalog manager: success
I20260620 18:35:17.869347 23432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: acquiring CA information for follower catalog manager: success
I20260620 18:35:17.871879 23432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88a72b4bfa384bc9b9a1a0058785b540: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:35:17.978626 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:17.969617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.97.60:33173)
0620 18:35:17.969824 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.969830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.975422 (+ 5592us) server_negotiation.cc:407] Connection header received
0620 18:35:17.977061 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.977067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.977158 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.977242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.978376 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39472: BlockingRecv error: recv got EOF from 127.0.0.1:39472 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:35:17.980541 23370 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:17.961086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.62:33241)
0620 18:35:17.961293 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.961301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.974934 (+ 13633us) server_negotiation.cc:407] Connection header received
0620 18:35:17.978957 (+ 4023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.978963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.979055 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.979129 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.980250 (+ 1121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43570: BlockingRecv error: recv got EOF from 127.0.0.1:43570 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:35:17.984031 23294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:17.976135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.97.61:32957)
0620 18:35:17.976342 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:17.976348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:17.981716 (+ 5368us) server_negotiation.cc:407] Connection header received
0620 18:35:17.982648 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:17.982654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:17.982757 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:17.982830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:17.983802 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52672: BlockingRecv error: recv got EOF from 127.0.0.1:52672 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:18.035662 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.018618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.97.61:32957)
0620 18:35:18.018809 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.018815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.027404 (+ 8589us) server_negotiation.cc:407] Connection header received
0620 18:35:18.027776 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.027784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.028077 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.028231 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.029069 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.030319 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.032442 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.033102 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.033210 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.033960 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.034699 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.034830 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.035098 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.035204 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:35:18.074548 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.065259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.0.97.60:33173)
0620 18:35:18.065469 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.065476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.072336 (+ 6860us) server_negotiation.cc:407] Connection header received
0620 18:35:18.072888 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.072894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.073050 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.073142 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.074260 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39488: BlockingRecv error: recv got EOF from 127.0.0.1:39488 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:35:18.126750 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.108641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.0.97.61:32957)
0620 18:35:18.108858 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.108865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.119551 (+ 10686us) server_negotiation.cc:407] Connection header received
0620 18:35:18.120081 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.120088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.120206 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.120281 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.121028 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.122228 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.124341 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.125065 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.125171 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.125829 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.125993 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.126062 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.126324 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.126430 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:35:18.170558 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.161484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.0.97.60:33173)
0620 18:35:18.161692 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.161699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.168546 (+ 6847us) server_negotiation.cc:407] Connection header received
0620 18:35:18.169061 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.169070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.169175 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.169255 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.170305 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39504: BlockingRecv error: recv got EOF from 127.0.0.1:39504 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:35:18.227299 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.209936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.0.97.61:32957)
0620 18:35:18.210114 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.210120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.220849 (+ 10729us) server_negotiation.cc:407] Connection header received
0620 18:35:18.221329 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.221336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.221424 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.221494 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.222180 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.223391 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.225218 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.225770 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.225864 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.226420 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.226559 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.226614 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.226861 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.226967 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 18:35:18.238292 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.236061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.97.60:33173)
0620 18:35:18.236300 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.236309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.236910 (+ 601us) server_negotiation.cc:407] Connection header received
0620 18:35:18.237123 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.237128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.237214 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.237326 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.238115 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39510: BlockingRecv error: recv got EOF from 127.0.0.1:39510 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:35:18.254545 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.249335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.97.61:32957)
0620 18:35:18.249563 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.249572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.249899 (+ 327us) server_negotiation.cc:407] Connection header received
0620 18:35:18.250111 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.250117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.250200 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.250304 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.250703 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.251468 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.252816 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.253302 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.253442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.253840 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.253938 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.254036 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.254175 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.254257 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 18:35:18.263779 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.261026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.0.97.60:33173)
0620 18:35:18.261308 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.261318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.262332 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 18:35:18.262545 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.262551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.262633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.262747 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.263459 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39514: BlockingRecv error: recv got EOF from 127.0.0.1:39514 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 18:35:18.280385 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.274673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.97.61:32957)
0620 18:35:18.274932 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.274945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.275438 (+ 493us) server_negotiation.cc:407] Connection header received
0620 18:35:18.275668 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.275672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.275740 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.275842 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.276246 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.277063 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.278345 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.278980 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.279115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.279563 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.279698 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.279809 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.279952 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.280033 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 18:35:18.291736 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.289407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.0.97.60:33173)
0620 18:35:18.289642 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.289652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.290154 (+ 502us) server_negotiation.cc:407] Connection header received
0620 18:35:18.290423 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.290429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.290537 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.290633 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.291429 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39518: BlockingRecv error: recv got EOF from 127.0.0.1:39518 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:35:18.323918 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.317466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.0.97.61:32957)
0620 18:35:18.317686 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.317696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.318341 (+ 645us) server_negotiation.cc:407] Connection header received
0620 18:35:18.318586 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.318590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.318658 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.318768 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.319179 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.320311 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.321956 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.322399 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.322552 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.323032 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.323155 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.323245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.323477 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.323589 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:35:18.332409 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.330178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.0.97.60:33173)
0620 18:35:18.330404 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.330414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.330938 (+ 524us) server_negotiation.cc:407] Connection header received
0620 18:35:18.331138 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.331144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.331236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.331346 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.332138 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39522: BlockingRecv error: recv got EOF from 127.0.0.1:39522 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:35:18.374145 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.367676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.0.97.61:32957)
0620 18:35:18.367959 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.367966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.368238 (+ 272us) server_negotiation.cc:407] Connection header received
0620 18:35:18.368414 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.368422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.368545 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.368700 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.369129 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.370462 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.371918 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.372681 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.372850 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.373208 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.373364 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.373471 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.373665 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.373758 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 18:35:18.382826 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.380982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.0.97.60:33173)
0620 18:35:18.381179 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.381186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.381478 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:35:18.381616 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.381624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.381737 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.381871 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.382562 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39536: BlockingRecv error: recv got EOF from 127.0.0.1:39536 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:35:18.455735 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.450456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.97.61:32957)
0620 18:35:18.450712 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.450720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.451063 (+ 343us) server_negotiation.cc:407] Connection header received
0620 18:35:18.451286 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.451291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.451406 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.451504 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.451924 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.452663 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.453970 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.454467 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.454582 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.454950 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.455050 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.455128 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.455338 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.455410 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 18:35:18.463795 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:18.461556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.0.97.60:33173)
0620 18:35:18.461785 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.461792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.462129 (+ 337us) server_negotiation.cc:407] Connection header received
0620 18:35:18.462436 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.462441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.462524 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.462631 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.463419 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39550: BlockingRecv error: recv got EOF from 127.0.0.1:39550 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:18.506748 23383 catalog_manager.cc:1259] Loaded cluster ID: f96f2a98634a4c89837c4916d67d636f
I20260620 18:35:18.506860 23383 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: loading cluster ID for follower catalog manager: success
I20260620 18:35:18.509346 23383 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: acquiring CA information for follower catalog manager: success
I20260620 18:35:18.511018 23383 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97e819ee17e54ff988c331eecc59c97f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:35:18.600837 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.595503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.0.97.61:32957)
0620 18:35:18.595769 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.595777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.596097 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:35:18.596399 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.596408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.596492 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.596593 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.597027 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.597807 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.599156 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.599605 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.599738 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.600133 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.600241 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.600346 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.600491 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.600571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:35:18.613757 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.607598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.0.97.60:33173)
0620 18:35:18.607850 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.607857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.608141 (+ 284us) server_negotiation.cc:407] Connection header received
0620 18:35:18.608321 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.608325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.608487 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.608600 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.609038 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.609831 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.611190 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.611725 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.611864 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.612385 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.612996 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.613113 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.613311 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.613444 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:18.629149 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.620462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.0.97.62:33241)
0620 18:35:18.621732 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.621738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.621761 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:18.621911 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.621919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.622175 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.623108 (+ 933us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.623314 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.624586 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.626450 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.627091 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.627239 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.627876 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.628422 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.628558 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.628679 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.628769 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":1009,"threads_started":1}
I20260620 18:35:18.630066 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.620693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.0.97.60:33173)
0620 18:35:18.620976 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.620986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.622325 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 18:35:18.622611 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.622620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.622720 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.622812 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.623757 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.624679 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.626817 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.627560 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.627694 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.628266 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.628417 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.628549 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.629665 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.629757 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:18.630369 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.622529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.97.61:32957)
0620 18:35:18.622786 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.622796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.623479 (+ 683us) server_negotiation.cc:407] Connection header received
0620 18:35:18.623743 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.623750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.623851 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.623956 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:18.624554 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.625644 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.628072 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.628738 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.628877 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.629472 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:18.629604 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:18.629701 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.629974 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.630066 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:35:18.634522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 23149
I20260620 18:35:18.635694 23211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:18.746273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 23149
W20260620 18:35:18.769603 23241 connection.cc:535] server connection from 127.0.0.1:52624 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:35:18.770255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 23223
I20260620 18:35:18.771498 23285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:18.881400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 23223
I20260620 18:35:18.902418 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu pid 23297
I20260620 18:35:18.903656 23359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:19.013343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcfg4Mf/build/asan/bin/kudu with pid 23297
2026-06-20T18:35:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4516 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9457 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366723 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST